Load test charts with standard deviation and percentile statistics
A method for real-time analysis of results from a load test performed on a target website includes calculating first-level aggregated test results within each of a plurality of load server instances that generate a load on the target website. The first-level aggregated test results are calculated from data points received by each of the load server instances from the target website. The first-level aggregated test results include a sum of the data points, a count of the number of the data points, a sum of squares of the data points, and an average of the data points. A standard deviation result (STDEV) is calculated and chart is generated on a display via a graphical user interface. The chart provides a visual representation of a performance metric for the load test based on the standard deviation result.
1. A computer-implemented method for real-time analysis of results obtained from a load test as the load test is being performed on a target website, comprising:
(a) computing first-level aggregated test results within each of a plurality of load server instances that generate a load on the target website, the first-level aggregated test results including first-level statistics computed from a set of data points received over a time period by each of the load server instances from the target website, the first-level statistics being computed by an embedded component within each of the load server instances, the first-level statistics including a sum of the data points, a count of the number of the data points, a sum of squares of the data points, and an average of the data points;
(b) sending the first-level aggregated test results from each of the load server instances to an associated one of a plurality of analytic server instances;
(c) aggregating, in one or more analytic server instances, all of the first-level aggregated test results computed by the load server instances resulting in second-level aggregated test results;
(d) aggregating all of the second-level aggregated test results resulting in a cumulative sum, a total count, a total sum of squares, and a cumulative average, of all of the data points across an entire data set;
(e) calculating a standard deviation result (STDEV) as
STDEV=SQRT[{( X 1 2 +X 2 2 + . . . +X N 2 )+ NA 2 −2 A ( X 1 +X 2 + . . . +X N )}/ N];
where N is the total count, A is the cumulative average, X 1 , X 2 . . . X N are the data points across the entire data set, (X 1 2 +X 2 2 + . . . +X N 2 ) is the total sum of squares, and (X 1 +X 2 + . . . +X N ) is the cumulative sum; and
(f) generating a chart on a display via a graphical user interface, the chart providing a visual representation of a performance metric for the load test based on the standard deviation result,
wherein steps (a)-(f) are each performed in real-time as the load test is being performed on the target website.
2. The computer-implemented method of claim 1 further comprising repeating steps (a)-(f) for a duration of the load test.
3. The computer-implemented method of claim 1 further comprising:
storing the second-level aggregated test results in a data storage instance.
4. The computer-implemented method of claim 3 further comprising communicating the standard deviation result to a browser application, the browser application providing an analytic dashboard with the graphical user interface.
5. The computer-implemented method of claim 1 wherein the chart is generated in real-time responsive to an input command of the graphical user interface.
6. The computer-implemented method of claim 1 wherein the performance metric comprises a confidence interval percentile (X).
7. The computer-implemented method of claim 1 further comprising calculating the confidence interval percentile (X) as
X ={( A +( Z *STDEV)};
where Z is a constant used for calculating a certain percentile.
8. The computer-implemented method of claim 7 wherein the certain percentile comprises either a 90th percentile, a 95th percentile, or a 98th percentile.
9. The computer-implemented method of claim 1 further comprising calculating a delta STDEV based on incremental changes occurring as the load test is running in real-time.
10. A non-transitory computer-readable storage medium encoded with computer instructions, which, when executed by a computer processor, operable to:
(a) compute first-level aggregated test results within each of a plurality of load server instances that generate a load on the target website, the first-level aggregated test results including first-level statistics computed from a set of data points received over a time period by each of the load server instances from the target website, the first-level statistics being computed by an embedded component within each of the load server instances, the first-level statistics including a sum of the data points, a count of the number of the data points, a sum of squares of the data points, and an average of the data points;
(b) send the first-level aggregated test results from each of the load server instances to an associated one of a plurality of analytic server instances;
(c) aggregating, in one or more analytic server instances, all of the first-level aggregated test results computed by the load server instances resulting in second-level aggregated test results:
(d) aggregating all of the second-level aggregated test results resulting in a cumulative sum, a total count, a total sum of squares, and a cumulative average, of all of the data points across an entire data set;
(e) calculating a standard deviation result (STDEV) as
STDEV=SQRT[{( X 1 2 +X 2 2 + . . . +X N 2 )+ NA 2 −2 A ( X 1 +X 2 + . . . +X N )}/ N];
where N is the total count, A is the cumulative average, X 1 , X 2 . . . X N are the data points across the entire data set, (X 1 2 +X 2 2 + . . . +X N 2 ) is the total sum of squares, and (X 1 +X 2 + . . . +X N ) is the cumulative sum; and
(f) generating a chart on a display via a graphical user interface, the chart providing a visual representation of a performance metric for the load test based on the standard deviation result,
wherein steps (a)-(f) are each performed in real-time as the load test is being performed on the target website.
11. The non-transitory computer readable storage medium of claim 10 wherein execution of the computer instructions is further operable to repeat steps (a)-(f) for a duration of the test load.
12. The non-transitory computer-readable storage medium of claim 10 wherein execution of the computer instructions is further operable to:
store the second-level aggregated test results in a data storage instance.
13. The non-transitory computer-readable storage medium of claim 12 wherein execution of the computer instructions is further operable to communicate the standard deviation result to a browser application, the browser application providing an analytic dashboard with the graphical user interface.
14. The non-transitory computer-readable storage medium of claim 10 wherein the chart is generated in real-time responsive to an input command of the graphical user interface.
15. The non-transitory computer-readable storage medium of claim 10 wherein the performance metric comprises a confidence interval percentile (X).
16. The non-transitory computer-readable storage medium of claim 10 wherein execution of the computer instructions is further operable to calculate the confidence interval percentile (X) as
X ={( A +( Z *STDEV)};
where Z is a constant used for calculating a certain percentile.
17. The non-transitory computer-readable storage medium of claim 16 wherein the certain percentile comprises either a 90th percentile, a 95th percentile, or a 98th percentile.
18. The non-transitory computer-readable storage medium of claim 10 wherein execution of the computer instructions is further operable to calculate a delta STDEV based on incremental changes occurring as the load test is running in real-time.