Analyzing time-series data in an automated application testing system
Methods and apparatus are described by which time-series data captured during the automated testing of software applications may be analyzed. Change-point detection is used to partition the time-series data, and an expected variance of data within each partition is determined. Because the partitioning of the test data provides a high level of confidence that the data points in a given partition conform to the same distribution, data points that represent meaningful changes in application performance can be more confidently and efficiently identified.
1. A platform, comprising one or more computing devices including one or more processors and memory configured to:
conduct a plurality of first test runs of one or more versions of a test for one or more versions of an application under test (AUT) by allocating computing resources;
receive first time-series data, the first time-series data including a data point representing a first performance metric for each of the first test runs;
partition the first time-series data into one or more partitions using change-point detection, each partition representing a corresponding range of the first time-series data belonging to a corresponding probabilistic distribution;
filter outliers of the data points in each partition, thereby resulting in remaining data points in each partition;
determine a range of expected behavior for the first performance metric for a first partition of the one or more partitions using the remaining data points of the first partition; and
evaluate a plurality of second test runs based on the range of expected behavior for the first partition.
2. The platform of claim 1 , wherein the one or more processors and memory of the one or more computing devices are further configured to:
receive second time-series data for the AUT, the second time-series data including a data point representing the first performance metric for each of the second test runs; and
identify one or more data points of the second time-series data that depart from the range of expected behavior for the first partition.
3. The platform of claim 2 , wherein the one or more processors and memory of the one or more computing devices are further configured to identify a change-point candidate from among the one or more data points of the second time-series data that depart from the range of expected behavior for the first partition.
4. The platform of claim 3 , wherein the one or more processors and memory of the one or more computing devices are further configured to:
identify a second partition of the second time-series data based on the change-point candidate;
determine a range of expected behavior for the first performance metric for the second partition; and
evaluate a plurality of third test runs based on the range of expected behavior for the second partition.
5. The platform of claim 1 , wherein the one or more processors and memory of the one or more computing devices are configured to filter the outliers of the data points in each partition using interquartile range filtering.
6. The platform of claim 1 , wherein the one or more processors and memory of the one or more computing devices are configured to determine the range of expected behavior for the first performance metric for the first partition by determining a standard deviation band based on the remaining data points.
7. The platform of claim 1 , wherein the computing resources include a plurality of virtual computing environment (VCE) instances, each VCE instance corresponding to a device operating system (OS).
8. The platform of claim 7 , wherein the device OS for a first VCE instance is a personal computer OS, and wherein the first VCE instance also corresponds to a web browser configured to interact with a corresponding instance of the AUT.
9. The platform of claim 8 , wherein the one or more processors and memory of the one or more computing devices are configured to conduct the plurality of first test runs by applying control commands to the web browser, and wherein the first time-series data represent a state of the web browser received from the web browser via an application programming interface associated with the web browser.
10. The platform of claim 7 , wherein the device OS for a first VCE instance is a mobile device OS, and wherein the first VCE instance also corresponds to a hardware device configured to operate with the mobile device OS.
11. The platform of claim 10 , wherein the first VCE instance also corresponds to a web browser configured to interact with a corresponding instance of the AUT.
12. The platform of claim 11 , wherein the one or more processors and memory of the one or more computing devices are configured to conduct the plurality of first test runs by applying control commands to the web browser, and wherein the first time-series data represent a state of the web browser received from the web browser via an application programming interface associated with the web browser.
13. The platform of claim 10 , wherein the AUT is a native application configured to operate with the mobile device OS, and wherein the first time-series data represent one or both of a state of the native application or a state of the mobile device OS.
14. The platform of claim 10 , wherein the first time-series data represent one or both of a state of the hardware device or a state of the mobile device OS.
15. The platform of claim 7 , wherein the device OS for a first VCE instance is a mobile device OS, and wherein the first VCE instance also corresponds to a mobile device emulator configured to emulate the mobile device OS.
16. The platform of claim 15 , wherein the first VCE instance also corresponds to a web browser configured to interact with a corresponding instance of the AUT.
17. The platform of claim 16 , wherein the one or more processors and memory of the one or more computing devices are configured to conduct the plurality of first test runs by applying control commands to the web browser, and wherein the first time-series data represent a state of the web browser received from the web browser via an application programming interface associated with the web browser.
18. The platform of claim 15 , wherein the AUT is a native application configured to operate with the mobile device OS, and wherein the first time-series data represent one or both of a state of the native application or a state of the mobile device OS.
19. The platform of claim 15 , wherein the first time-series data represent one or both of a state of the mobile device emulator or a state of the mobile device OS.
20. A computer program product, comprising one or more non-transitory computer-readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more computing devices, the computer program instructions cause the one or more computing devices to:
conduct a plurality of first test runs of one or more versions of a test for one or more versions of an application under test (AUT) by allocating computing resources;
receive first time-series data, the first time-series data including a data point representing a first performance metric for each of the first test runs;
partition the first time-series data into one or more partitions using change-point detection, each partition representing a corresponding range of the first time-series data belonging to a corresponding probabilistic distribution;
filter outliers of the data points in each partition, thereby resulting in remaining data points in each partition;
determine a range of expected behavior for the first performance metric for a first partition of the one or more partitions using the remaining data points of the first partition; and
evaluate a plurality of second test runs based on the range of expected behavior for the first partition.
21. The computer program product of claim 20 , wherein the computer program instructions are further configured to cause the one or more computing devices to:
receive second time-series data for the AUT, the second time-series data including a data point representing the first performance metric for each of the second test runs; and
identify one or more data points of the second time-series data that depart from the range of expected behavior for the first partition.
22. The computer program product of claim 21 , wherein the computer program instructions are further configured to cause the one or more computing devices to identify a change-point candidate from among the one or more data points of the second time-series data that depart from the range of expected behavior for the first partition.
23. The computer program product of claim 22 , wherein the computer program instructions are further configured to cause the one or more computing devices to:
identify a second partition of the second time-series data based on the change-point candidate;
determine a range of expected behavior for the first performance metric for the second partition; and
evaluate a plurality of third test runs based on the range of expected behavior for the second partition.
24. The computer program product of claim 20 , wherein the computer program instructions are configured to cause the one or more computing devices to filter the outliers of the data points in each partition using interquartile range filtering.
25. The computer program product of claim 20 , wherein the computer program instructions are configured to cause the one or more computing devices to determine the range of expected behavior for the first performance metric for the first partition by determining a standard deviation band based on the remaining data points.