Distributed test suite tools
Reliably testing distributed systems in which multiple components interact, including: provisioning resources by a test leader, directing components to perform actions required by test suites, and reporting results. Provisioning resources might be batched in advance, and might overlap resources for multiple test suites. Provisioning and directing might occur on an as-needed basis, might be performed concurrently, and might be reordered for relatively better performance. Directing actions might occur on command by test leaders, might occur by synchronization of resources, or might occur by resources requesting permission to proceed. Resources might act only upon direction by test leaders, or might act on their own initiative when synchronized. Results might be reported to the test leader for comparison, maintained in a database for access by the test leader, or compared by resources on their own initiative in response to instructions from the test leader.
1. A system including
a distributed computing system operating using a distributed operating system, the distributed computing system including a plurality of test leaders including computing devices using instructions, each independently performing one or more of its own test suites, the test leaders each independently provisioning one or more resources for its own test suites, wherein the test leaders exchange information with respect to one or more of: test suites, resource designations, or test suite instructions;
the computing devices using instructions directing those test leaders to send instructions to resources in the distributed computing system to perform operations according to those test suites, the operations according to those test suites including one or more operations of those resources using the distributed operating system and at least potentially interfering when performed concurrently;
the computing devices using instructions directing those test leaders to access one or more results from those operations, to compare results from those operations with one or more results indicated by those test suites, and to access results from those operations;
wherein the distributed computing system includes messages from those resources to those test leaders.
2. A system as in claim 1 ,
including instructions at those test leaders to select one or more test suites matching suite tags for which test suites are to be performed.
3. A system as in claim 1 ,
including instructions at those test leaders to provision resources for a first test suite, conduct that first test suite, subsequently provision resources for a second test suite, and independently conduct that second test suite;
wherein when resources for the first test suite and the second test suite conflict, the test leader orders the first and the second test suite to be performed in response to a measure of performance.
4. A system as in claim 1 ,
wherein those operations according to those test suites are performed with-in a time to send messages between those resources and those test leaders.
5. A system as in claim 1 ,
wherein those instructions directing resources to perform operations include instructions to distribute information identifying those operations before those operations are to be performed;
wherein those resources include devices that identify those operations in response to that information; and
wherein those resources perform those operations at those times specified in that information.
6. A system as in claim 5 ,
wherein that information includes information identifying specific times when those operations are to be performed.
7. A system as in claim 1 ,
clocks at each of those test leaders and at each of those resources, the clocks at each test leader being synchronized with the resources instructed by that test leader;
wherein instructions from each test leader directing resources to perform operations are sent before those operations are to be performed.
8. A system as in claim 7 ,
including instructions directing those resources to send synchronization messages to those test leaders that those operations are about to be performed; and
including instructions directing those test leaders responding those synchronization messages from those resources.
9. A system as in claim 1 ,
including instructions directing those test leaders to provision one or more resources for a plurality of test suites and direct those resources to perform those test suites; and
including instructions directing those test leaders to perform those actions of provisioning and directing for at least a portion of those test suites, repeatedly until resources for all those test suites have been provisioned.
10. A system as in claim 1 ,
including instructions directing those resources to perform those test suites concurrently.
11. A system as in claim 1 ,
including instructions directing those test leaders to provision one or more resources in response to a measure of performance of those test suites.
12. A system as in claim 1 ,
including instructions directing those test leaders to provision one or more resources in response to those test suites, wherein use of those resources overlaps with respect to those test suites; and
including instructions directing those test leaders to provision one or more resources before sending messages to those resources to perform operations.
13. A system as in claim 1 ,
wherein those test suites include configuration information interpretable as indicating
one or more resources to be used in conjunction with those test suites;
one or more operations to be performed in conjunction with those test suites;
one or more tags distinguishing types of test suites.
14. A system as in claim 13 ,
wherein those resources include one or more of: data structures, processors, storage devices;
the data structures including one or more of: databases, locks, shared directories, test results.
15. A system as in claim 1 ,
wherein multiple test leaders each include instructions disposed to be independently performed on separate devices:
to receive messages from those resources including information with respect to one or more results from those operations,
to access at least some of those results,
to compare results from those operations with one or more results indicated by those test suites, and
to access results from those operations; and
the messages from those resources to those test leaders include information with respect to one or more results from those operations.