System and method of test iteration via property chaining
A new test control structure improves on constructing complex test sequences in a scripting language. The new test control structure iterates over two or more arbitrary values of a test attribute, such as a networking protocol parameter applied at OSI layers 2-7.
1. A method used by a test harness during a stress test of a system under test, including:
bounding a set of test commands with a transitory iterator command and a transitory iteration-verifier command, wherein test commands are passed to a test harness that is coupled to a system under test and the test commands are expressed in a declarative command language;
wherein the transitory iterator command and the transitory iteration-verifier command are active when they are executed and are inactive when the bounded test commands are active;
performing a sequence of network traffic stress tests in which test commands cause the test harness to send test traffic messages to the system under test via one or more network ports and measure performance of the system under test;
wherein the transitory iterator command and iteration-verifier command combine to implement a stability backoff test of stress levels;
beginning in an expected failure range that is above an approximately measured pass/fail threshold;
the transitory iterator command backing off an applied stress applied at an applied stress level by the bounded test commands;
repeating the applied stress a predetermined number of times, which is more than once for the applied stress level;
the iteration-verifier command, evaluating a pass/fail result for an iteration of the applied stress level, and accumulating a pass/fail rate for results of iterations at the applied stress level; and
evaluating the pass/fail rate accumulated, after the predetermined number of times iterating at the applied stress level, against a predetermined pass rate, and scoring the applied stress level as passed or failed;
the transitory iterator command terminating the test when the predetermined pass rate has been achieved at a predetermined number of applied stress levels as the transitory iterator command progressively backs off the applied stress level; and
one or more result commands outside the bounded test commands reporting test results.
2. The method of claim 1 , wherein expressing test commands in a declarative command language, includes an end user modifying exposed data to control test process behavior.
3. The method of claim 1 , wherein transitory commands are active during execution on the test harness.
4. The method of claim 1 , wherein transitory commands are inactive when not being executed on the test harness.
5. The method of claim 1 , wherein measuring performance of the system under test includes measuring frame dropping by a pass-through SUT.
6. The method of claim 1 , wherein measuring performance of the system under test includes measuring command response by a SUT that responds to configuration commands during the test.
7. The method of claim 1 , wherein implementing a stability backoff test of stress levels includes testing through-put capacity.
8. The method of claim 1 , wherein implementing a stability backoff test of stress levels includes testing command processing capacity while under load.
9. The method of claim 1 , wherein beginning in an expected failure range that is above an approximately measured pass/fail threshold includes chaining ramp-up test determining the approximately measured pass/fail threshold.
10. The method of claim 1 , wherein beginning in an expected failure range that is above an approximately measured pass/fail threshold includes binary search test determining the approximately measured pass/fail threshold.
11. The method of claim 1 , wherein beginning in an expected failure range that is above an approximately measured pass/fail threshold includes backoff resolution has a resolution at least ten times as precise as the approximately measured pass/fail threshold.
12. The method of claim 1 , wherein the transitory iterator command backing off an applied stress applied by the bounded test commands includes parameter chaining.
13. The method of claim 1 , including the iteration-verifier command causing a query against databases collecting results of each iteration at the applied stress level.
14. The method of claim 13 , wherein the iteration-verifier command evaluates a pass/fail result for an iteration of the applied stress level from a response to the query.
15. The method of claim 1 , wherein the transitory iterator command progressively backs off the applied stress level using a specified backoff increment.
16. A test system for conducting stress testing of a system under test, including:
A test harness coupleable to the system under test; and
A test sequence processor that:
bounds a set of test commands with a transitory iterator command and a transitory iteration-verifier command, wherein test commands are passed to a test harness that is coupled to a system under test and the test commands are expressed in a declarative command language;
wherein the transitory iterator command and the transitory iteration-verifier command are active when they are executed and are inactive when the bounded test commands are active;
performs a sequence of network traffic stress tests in which test commands cause the test harness to send test traffic messages to the system under test via one or more network ports and measure performance of the system under test;
wherein the transitory iterator command and iteration-verifier command combine to implement a stability backoff test of stress levels;
begins in an expected failure range that is above an approximately measured pass/fail threshold;
the transitory iterator command backing off an applied stress applied at an applied stress level by the bounded test commands;
repeats the applied stress a predetermined number of times, which is more than once for the applied stress level;
the iteration-verifier command, evaluating a pass/fail result for an iteration of the applied stress level, and accumulating a pass/fail rate for results of iterations at the applied stress level; and
evaluates the pass/fail rate accumulated, after the predetermined number of times iterating at the applied stress level, against a predetermined pass rate, and scoring the applied stress level as passed or failed;
the transitory iterator command terminating the test when the predetermined pass rate has been achieved at a predetermined number of applied stress levels as the transitory iterator command progressively backs off the applied stress level; and
one or more result commands outside the bounded test commands reporting test results.
17. A tangible non-volatile computer readable medium that stores computer instructions that, when run on hardware, implement a stress test of a system under test, which instructions implement actions including:
bounding a set of test commands with a transitory iterator command and a transitory iteration-verifier command, wherein test commands are passed to a test harness that is coupled to a system under test and the test commands are expressed in a declarative command language;
wherein the transitory iterator command and the transitory iteration-verifier command are active when they are executed and are inactive when the bounded test commands are active;
performing a sequence of network traffic stress tests in which test commands cause the test harness to send test traffic messages to the system under test via one or more network ports and measure performance of the system under test;
wherein the transitory iterator command and iteration-verifier command combine to implement a stability backoff test of stress levels;
beginning in an expected failure range that is above an approximately measured pass/fail threshold;
the transitory iterator command backing off an applied stress applied at an applied stress level by the bounded test commands;
repeating the applied stress a predetermined number of times, which is more than once for the applied stress level;
the iteration-verifier command, evaluating a pass/fail result for an iteration of the applied stress level, and accumulating a pass/fail rate for results of iterations at the applied stress level; and
evaluating the pass/fail rate accumulated, after the predetermined number of times iterating at the applied stress level, against a predetermined pass rate, and scoring the applied stress level as passed or failed;
the transitory iterator command terminating the test when the predetermined pass rate has been achieved at a predetermined number of applied stress levels as the transitory iterator command progressively backs off the applied stress level; and
one or more result commands outside the bounded test commands reporting test results.