Unified test automation system
An automated testing framework to coordinate functions among code management, build management, automated test, resource reservation, artifact repositories and team communication subsystems. In one embodiment specific to software development, software developers check new code into a code management subsystem, a development project tracking system, or other tools that automatically generate a build. Test suites are then automatically selected and executed based on a scope of the project or code that was built. This scope can include such attributes as what portion of the software was built, the purpose (objective) of the build, the maturity of developing that build, and so forth. In one implementation, label directives may be used to associate build scope to test suites. During or after the automated tests, other actions may then be automatically triggered, to store test results, inform the development team, stored data integrated with test definition and results, or update an artifact repository.
1. A programmatic, automatically executing method performed by one or more data processors, the method for executing a test suite comprising:
(a) receiving, from a code build automation system via a communication network, a notification of a code push event;
(b) triggering a code build;
(c) receiving, from the code build automation system via a communication network, the code build and a scope of the code build;
(d) determining a test suite to execute against the code build by matching one or more label directives associated with the scope of the code build;
(e) executing the test suite against a system under test;
(f) if the test suite passes,
i. updating test execution status to pass;
(g) else if the test suite fails,
i. updating test execution status to fail;
ii. recording a defect; and
(h) storing results of the test execution.
2. The method of claim 1 wherein a build maturity attribute determines the scope.
3. The method of claim 1 wherein a purpose of the build determines the scope.
4. The method of claim 1 wherein the test suite comprises one or more test cases selected from a group consisting of must pass, sanity, regression, functional, performance, scalability, reliability, or endurance tests.
5. The method of claim 1 wherein the label directives are associated with some attribute of the test suite.
6. The method of claim 1 additionally comprising:
(j) automatically reserving resources for executing the test suite;
(k) releasing resources before test suite completes if at least one test case fails.
7. The method of claim 1 additionally comprising;
(k) ensuring a configuration for the system under test is updated before executing the test suite.
8. The method of claim 1 additionally comprising:
(l) performing a security integrity check.
9. The method of claim 1 additionally comprising:
(m) copying a definition of the test suite to a clone copy in a project management system.
10. The method of claim 9 additionally comprising;
(n) logging results of the test execution only to the clone copy and not to the definition of the test suite.
11. The method of claim 1 wherein the steps of detecting code push and triggering a build are performed in an integrated build management system.
12. The method of claim 1 wherein the test suite specifies execution dependencies among two or more test cases.
13. The method of claim 1 additionally comprising managing a defect lifecycle by one or more of:
i. automatically logging defect details to log file;
ii. entering defects into a defect management system;
iii. assigning and tracking defect identifiers;
iv. updating defect hit count statistics; and
v. re-opening a previously resolved defect upon rediscovering the same defect.
14. The method of claim 1 executed as a service.
15. The method of claim 1 additionally comprising:
performing one or more defect management lifecycle steps.
16. A test automation system for testing a data processing system under test (SUT), the test automation system comprising one or more processors having one or more non-transitory memory devices coupled thereto, the one or more non-transitory memory devices storing program code that when executed by the one or more processors cause the one or more processors to provide:
an interface via a communications network to a code management system;
an interface via a communications network to a resource reservation
system, for requesting reservation of resources within the system under test;
an interface via a communication network to a project management system;
a test suite execution engine, for executing the program code for:
(a) receiving notification of a code build event from the code management system and a scope of the code build;
(b) retrieving a test suite from the project management system;
(c) determining a test suite to execute against the code build by matching one or more label directives associated with the scope of the code build;
(d) requesting reservation of resources from the resource reservation system;
(e) executing the test suite against the system under test;
(f) if the test suite passes,
i. updating a test execution status to pass;
(g) else if the test suite fails
i. updating a test execution status to fail;
ii. recording a defect; and
(h) storing results of the test execution.
17. The system of claim 16 wherein a build maturity attribute determines the scope.
18. The system of claim 16 wherein a purpose of the build determines the scope.
19. The system of claim 16 wherein the test suite comprises one or more test cases selected from a group consisting of must pass, sanity, smoke, regression,
functional, performance, scalability, reliability, or endurance tests.
20. The system of claim 16 additionally comprising:
(l) an interface to a project management system; and
(m) wherein the program code is further for requesting the project management system to store a clone copy of the test suite before executing the test suite.
21. The system of claim 20 additionally wherein the program code is further for requesting the project management system to associate a result of the test execution only with the clone copy and not with the test suite.