Architecture, method and system for live testing in a production environment
There is provided an architecture, methods and a system for live testing in a production environment. The architecture comprises a platform independent Test Planner for generating a test package in response to receiving an event. Generating a test package comprises selecting test goals, generating a test suite and generating a test plan. The architecture comprises a platform dependent Test Execution Framework (TEF) for executing the test package in an environment serving live traffic. Executing the test package comprises initializing the test plan, starting the test plan and reporting the successful completion of the test plan, reporting the suspension of the test plan and waiting for further instructions, or reporting a failure of the test plan and executing a corresponding contingency plan.
1. An architecture for live testing, comprising:
a platform independent Test Planner for generating a test package in response to receiving an event, wherein generating the test package comprises:
selecting test goals corresponding to the event;
generating a test suite comprising test cases and test design techniques selected to achieve the test goals; and
generating a test plan comprising partially ordered Test Suite Item runs (TSIs), wherein each TSI combines a test configuration with at least one test suite item;
wherein Unified Modeling Language (UML) Testing Profile (UTP) is used as a modeling tool for at least the test goals, the test cases, the test design techniques, the test suite and the test plan; and
wherein the test goals that are associated with test cases are modeled as UTP TestRequirement and the test goals that are associated with test design techniques are modeled as UTP TestObjective; and
a platform dependent Test Execution Framework (TEF) for executing the test package in an environment serving live traffic;
wherein, during execution of the test package, the TEF creates a hierarchical structure of the test plan, creates an instance of a finite state machine (FSM), wherein the instance of the FSM moves through states in the FSM while exchanging messages with parent and child instances, and destroys the FSM when a final state is reached.
2. The architecture of claim 1 , further comprising updating a test repository using information collected about each executed test case.
3. The architecture of claim 1 , wherein generating the test suite is based on a runtime state of a system to be tested and wherein information about the selected test cases and about the selected test design techniques is complemented with appropriate test isolation methods and contingency plans to minimize any impact on live traffic.
4. The architecture of claim 3 , wherein the test cases and test design techniques are stored in a test repository, the test repository comprising information including mappings between the test goals, the test cases and the test design techniques.
5. The architecture of claim 1 , wherein the event comprises any one of: a periodic event, a system reconfiguration or change, an addition of a new test goal to a test repository, and a test request; and wherein the test request comprises any one of: an aggregation of periodic events, a set of test goals to achieve, a set of fault-revealing test goals, or a set of test cases.
6. The architecture of claim 1 , wherein the test plan is expressed as a TestExecutionSchedule that runs at least one UTP TestCase; and wherein a UTP TestCase consists of one or more test cases or test design techniques provided by a vendor or a developer.
7. The architecture of claim 1 , wherein failures are detected during execution of the test package as UTP verdict types.
8. The architecture of claim 7 , wherein the UTP verdict types comprise:
UTP verdicts None, PASS, FAIL, or INCONCLUSIVE which are deemed as successful test execution;
UTP verdict ERROR corresponding to a test execution failure of undetermined origin; and
UTP customized verdicts corresponding to identified test execution failures.
9. The architecture of claim 1 , wherein executing a test package on a live system serving live traffic comprises running the test suite items according to the test plan comprising the partially ordered TSIs, wherein each TSI combines a test configuration with at least one test suite item.
10. The architecture of claim 1 , wherein executing the test package on a live system serving live traffic comprises:
initializing a test plan;
starting the test plan, the test plan starting Test Suite Item runs (TSIs) of a test suite according to a provided ordering, the test plan being able to invoke one or more TSIs, for sequential or parallel execution;
upon successful completion of all TSIs, tearing down the test plan and reporting the successful completion of the test plan;
upon suspension of a TSI, reporting the suspension of the TSI and waiting for further instructions; and
upon failure of a TSI with an ERROR verdict, reporting a failure of the test plan and executing a corresponding contingency plan.