Unified test automation system
An automated testing framework to coordinate functions among code management, build management, automated test, resource reservation, artifact repositories and team communication subsystems. In one embodiment specific to software development, software developers check new code into a code management subsystem, a development project tracking system, or other tools that automatically generate a build. Test suites are then automatically selected and executed based on a scope of the project or code that was built. This scope can include such attributes as what portion of the software was built, the purpose (objective) of the build, the maturity of developing that build, and so forth. In one implementation, label directives may be used to associate build scope to test suites. During or after the automated tests, other actions may then be automatically triggered, to store test results, inform the development team, stored data integrated with test definition and results, or update an artifact respository.
1. A method for executing a test suite within one or more processors and one or more memories coupled to the one or more processors, the one or memories storing program code configured to be run by the one or more processors, the method further comprising:
(a) receiving, from a code build automation system via a communication network, notification of a code push event;
(b) in response to the code push event, triggering a code build;
(c) receiving, from the code build automation system via a communication network, the code build and a scope of the code build, the scope including at least one attribute of the code build;
(d) automatically determining a test suite from among multiple test suites to execute against the code build, each of the multiple test suites having at least one label directive, by matching one or more of the label directives against the at least one attribute of the code build;
(e) executing the automatically determined test suite against a system under test;
(f) if the automatically determined test suite passes,
i. updating test execution status to pass;
(g) else if the automatically determined test suite fails,
i. updating test execution status to fail;
ii. recording a defect; and
(h) storing results of the test execution.
2. The method of claim 1 wherein the at least one attribute is a build maturity attribute.
3. The method of claim 1 wherein the at least one attribute is a purpose of the code build.
4. The method of claim 1 wherein at least one of the test suites further comprises one or more test cases selected from a group consisting of must pass, sanity, smoke, regression, functional, performance, scalability, reliability, or endurance tests.
5. The method of claim 1 wherein the label directives are associated with some attribute of the automatically determined test suite.
6. The method of claim 1 additionally comprising:
(j) automatically reserving resources for executing the automatically determined test suite;
(k) releasing resources before the automatically determined test suite completes if at least one test case fails.
7. The method of claim 1 additionally comprising;
(k) ensuring a configuration for the system under test is updated before executing the automatically determined test suite.
8. The method of claim 1 additionally comprising:
(l) performing a security integrity check.
9. The method of claim 1 additionally comprising:
(m) copying a definition of the automatically determined test suite to a clone copy in a project management system.
10. The method of claim 9 additionally comprising;
(n) logging results of the test execution only to the clone copy and not to the definition of the automatically determined test suite.
11. The method of claim 1 wherein the steps of detecting code push and triggering a code build are performed in an integrated build management system.
12. The method of claim 1 wherein the automatically determined test suite specifies execution dependencies among two or more test cases.
13. The method of claim 1 additionally comprising managing a defect lifecycle by one or more of:
i. automatically logging defect details to log file;
ii. entering defects into a defect management system;
iii. assigning and tracking defect identifiers;
iv. updating defect hit count statistics; and
v. re-opening a previously resolved defect upon rediscovering the same defect.
14. The method of claim 1 executed as a service.
15. The method of claim 1 additionally comprising:
performing one or more defect management lifecycle steps.
16. A test automation system for testing a data processing system under test (SUT), the test automation system comprising:
an interface via a communications network to a code management system;
an interface via a communications network to a resource reservation system, for requesting reservation of resources within the system under test;
an interface via a communication network to a project management system;
a test suite execution engine, the test suite execution engine comprising one or more processors and one or more memories coupled to the one or more processors, the one or more memories for storing program code configured to be executed by the processor, the program code for:
(a) receiving notification of a code build event from the code management system, the code build including a scope of the code build, the scope including at least one attribute of the code build;
(b) retrieving multiple test suites from the project management system, each of the multiple test suites having at least one label directive;
(c) automatically determining a test suite from among the multiple test suites to execute against the code build, by matching one or more of the label directives against the at least one attribute of the code build;
(d) requesting reservation of resources from the resource reservation system;
(e) automatically executing the automatically determined test suite against the system under test;
(f) if the automatically determined test suite passes,
i. updating a test execution status to pass;
(g) else if the automatically determined test suite fails
i. updating a test execution status to fail;
ii. recording a defect; and
(h) storing results of the test execution.
17. The system of claim 16 wherein the at least one attribute is a build maturity attribute.
18. The system of claim 16 wherein the at least one attribute is a purpose of the build.
19. The system of claim 16 wherein the automatically determined test suite comprises
one or more test cases selected from a group consisting of must pass, sanity, smoke, regression,
functional, performance, scalability, reliability, or endurance tests.
20. The system of claim 16 additionally comprising:
(l) an interface to a project management system; and
(m) wherein the program code is further for requesting the project management system to store a clone copy of the automatically determined test suite before executing the automatically determined test suite.
21. The system of claim 20 additionally wherein the program code is further for requesting the project management system to associate a result of the test execution only with the clone copy and not with the automatically determined test suite.
22. The system of claim 16 wherein the automatically determined test suite specifies execution dependencies among two or more test cases.
23. The system of claim 22 wherein the execution dependencies comprise one or or more of a failure to reserve a resource, non-operability of a hardware or software component, or some other execution error.
24. The method of claim 12 wherein the execution dependencies comprise one or or more of a failure to reserve a resource, non-operability of a hardware or software component, or some other execution error.