Error tolerant modular testing of services
Methods of error-tolerant modular testing of services are described, wherein an ordered list of test module identifiers is built in an error stack for the purposes of structured state teardown following the occurrence of an error during testing of services (i.e., network or other.) The error that triggers the teardown may comprise any error or more particularly an error not among a predetermined list of acceptable errors, the occurrence of which should not cause the cessation of services testing. Upon the occurrence of a triggering error, the test modules associated with the test module identifiers are executed in a reverse order to that which the test module identifiers were added to the error stack, effecting a structured state teardown.
1. Method of error-tolerant modular testing of services, comprising the steps of:
sequentially populating an error stack with a test module identifier associated with one of a sequence of executed state-changing test modules operating upon a device interacting with a service;
upon the occurrence during execution of the sequence of state-changing test modules of a triggering error, running in a reverse order the test modules identified in the error stack so as to cause a structured teardown of the test sequence; and
repeating the previous steps until the sequential execution of test modules is complete or a triggering error is encountered.
2. The method of claim 1 , wherein:
each of the test modules contains an indication of acceptable errors; and
the triggering error comprises an error not among the acceptable errors.
3. The method of claim 1 , wherein the device is an active network probe in communications with a network, the service is a network service, and the sequence of state-changing tests operate to test the QoS of the network service.
4. The method claim 1 , further comprising the step of creating the error stack.
5. The method of claim 1 , further comprising the step of executing one or more non-state-changing tests interspersed among the sequence of executed state-changing tests without alteration of the error stack.
6. The method of claim 1 , further comprising the step of acquiring data relevant to the execution of each test module.
7. The method of claim 6 , further comprising the step of recording the acquired data prior to the occurrence of the triggering error.
8. The method of claim 7 , wherein the data comprises a pass/fail message indicating the success/failure of that test, indications of when during the test sequence an error occurred, and/or metrics related to the performance of the service.
9. The method of claim 1 , wherein the sequentially populating step further comprises pushing the test identification information of each test onto the error stack prior to starting that test.
10. The method of claim 1 , wherein the running in reverse order step further comprises popping the test module identifier of each successfully executed test module from the error stack.
11. A computer-readable medium storing a program which when executed by a computer causes the computer to execute the functions comprising:
sequentially populating an error stack witt a test module identifier associated with one of a sequence of executed state-changing test modules operating upon a device interacting with a service;
upon the occurrence during execution of the sequence of state-changing test modules of a triggering error, running in a reverse order the test modules identified in the error stack so as to cause a structured teardown of the test sequence; and
repeating the previous steps until the sequential execution of test modules is complete or a triggering error is encountered.
12. The medium of claim 11 , wherein:
each of the test modules contains an indication of acceptable errors; and
the triggering error comprises an error not among the acceptable errors.
13. The medium of claim 11 , wherein the device is an active network probe in communications with a network, the service is a network service, and the sequence of statechanging tests operate to test the QoS of the network service.
14. The medium of claim 11 , further comprising the step of creating the error stack.
15. The medium of claim 11 , wherein the functions further comprise executing one or more non-state-changing tests interspersed among the sequence of executed state-changing tests without altering the error stack.
16. The medium of claim 11 , wherein the functions further comprise acquiring data relevant to the execution of each test module.
17. The medium of claim 16 , wherein the data comprises a pass/fail message indicating the success/failure of that test, indications of when during the test sequence an error occurred, and/or metrics related to the performance of the service.
18. The medium of claim 16 , wherein the functions further comprise recording the acquired data prior to the occurrence of the triggering error.
19. The medium of claim 11 , wherein the sequentially populating function further comprises pushing the test identification information of each test onto the error stack before starting that test.
20. The medium of claim 11 , wherein the running in reverse order function further comprises popping the test module identifier of each successfully executed test module from the error stack.