Techniques for distributing tests and test suites across a network
Reliably testing distributed systems in which multiple components interact, including: provisioning resources by a test leader, directing components to perform actions required by test suites, and reporting results. Provisioning resources might be batched in advance, and might overlap resources for multiple test suites. Provisioning and directing might occur on an as-needed basis, might be performed concurrently, and might be reordered for relatively better performance. Directing actions might occur on command by test leaders, might occur by synchronization of resources, or might occur by resources requesting permission to proceed. Resources might act only upon direction by test leaders, or might act on their own initiative when synchronized. Results might be reported to the test leader for comparison, maintained in a database for access by the test leader, or compared by resources on their own initiative in response to instructions from the test leader.
1. A method, including steps of
at one or more test leaders in a distributed computing system, said test leaders provisioning resources for one or more test suites, wherein said test suites compete for said resources to be provisioned by said test leaders in said distributed computing system;
at said test leaders, directing said resources to perform operations according to those test suites, said operations according to those test suites including one or more operations of communicating using said distributed computing system, and said operations at least potentially interfering when performed concurrently;
at said test leaders, accessing one or more results from said operations;
comparing results from said operations with one or more results indicated by said test suites;
at said test leaders, accessing results from said operations;
including steps of sending messages from said resources to said test leaders;
wherein said steps of comparing are performed at said test leaders.
2. A method as in claim 1 ,
including steps, at resources of sending messages to said test leaders on their own initiative, providing results to said test leaders when said results are available.
3. A method, as in claim 1 ,
including steps, at said test leaders, of sending messages to said resources directing them to provide results to said test leaders.
4. A method, as in claim 1 ,
including steps of sending messages from said resources to said test leaders; and
wherein said steps of comparing are performed at said test leaders.
5. A method as in claim 1 , wherein,
said operations are performed in relative time proximity, said relative time proximity being substantially within a time to send messages amount said resources.
6. A method as in claim 1 , wherein,
said steps of directing said resources to perform operations include steps of distributing information interpretable as indicating said operations before said operations are to be performed;
at said resources, interpreting said distributed information; and
at said resources, performing said operations at times said test suites indicate they are to be performed.
7. A method as in claim 1 , wherein
said steps of directing said resources to perform operations include steps of
sending messages to said resources substantially at times when said operations are to be performed.
8. A method as in claim 1 , wherein
said steps of directing said resources to perform operations include steps of
synchronizing said resources with said test leader; and
sending one or more messages to said resources substantially before said operations are to be performed.
9. A method as in claim 8 , wherein
said steps of synchronizing include steps of
at said resources, sending ore or more messages to said test leaders indicating that said operations are about to be performed;
at said test leaders, responding to said messages to said test leaders.
10. A method as in claim 1 , wherein
said steps of provisioning and directing include steps of
provisioning said resources for a portion of said test suites, and directing said resources to perform a perform a portion of said operations; and
repeating said steps of provisioning and directing portions, until substantially all of said test suites are substantially completed.
11. A method as in claim 1 , wherein
said steps of provisioning and directing include steps of
provisioning said resources and directing said resources for a plurality of test suites to be performed substantially concurrently.
12. A method as in claim 1 , wherein
said steps of provisioning resources include steps of
ordering said steps of provisioning in response to a measure of performance of said test suites.
13. A method as in claim 1 , wherein
said steps of provisioning resources include steps of
provisioning said resources in response to a plurality of test suites, wherein said resources overlap within said plurality of test suites;
wherein said steps of provisioning are performed substantially before directing any said resources to perform operations.
14. A method as in claim 1 , wherein said test suites include information indicating
one or more resources to be used in conjunction with said test suites;
one or more operations to be performed in conjunction with said test suites;
one or more tags distinguishing types of said test suites.
15. A method as in claim 14 , wherein
said resources include one or more of:
data structures, processors, storage devices;
said data structures including one or more of: databases, locks, shared directories, test results.
16. A physical medium including information accessible by a computing device
said information being interpretable by said computing device as a set of instructions directing an element to
at one or more test leaders in a distributed operating system, disposing said test leaders to provision resources for one or more test suites, wherein said test suites compete for said resources to be provisioned by said test leaders in said distributed computing system;
at said test leaders, directing said resources to perform operations according to those test suites, said operations according to those test suites including one or more operations of communicating using said distributed computing system, and said operations at least potentially interfering when performed concurrently;
at said test leaders, access one or more results from said operations;
at said resources, send messages to said test leaders on their own initiative, providing results to said test leaders at times when said results are available;
send messages from said resources to said test leaders,
wherein said instructions to compare results are performed at said test leaders.
17. A physical medium as in claim 16 , wherein
said information interpretable as a set of instructions includes information interpretable as instructions to
compare results from said operations with one or more results indicated by said test suites; and
at said test leaders, access results from said operations.
18. A physical medium as in claim 17 , including instructions to
at said test leaders, send messages to said resources directing them to provide results to said test leaders.
19. A physical medium as in claim 16 , wherein
said instructions to direct said resources to perform operations include instructions to
send messages to said resources substantially at times when said operations are to be performed.
20. A physical medium as in claim 16 , wherein
said instructions to direct said resources to perform operations include instructions to
synchronize said resources with one or more test leaders; and
send one or more messages to said resources substantially at times when said operations are to be performed.
21. A physical medium as in claim 16 , wherein
said instructions to direct said resources to perform operations include instructions to
synchronize said resources with one or more test leaders; and
send one or more messages to said resources substantially before said operations are to be performed.
22. A physical medium as in claim 21 , wherein
said instructions to synchronize include instructions to
at said resources, send one or more messages to said test leaders indicating that said operations are about to be performed;
at said test leaders, respond to said messages to said test leaders.
23. A physical medium as in claim 16 , wherein
said instructions to provision and direct include instructions to
provision said resources for a portion of said test suites, and direct said resources to perform a portion of said operations; and
repeat said instruction to provision and direct portions, until substantially all of said test suites are substantially completed.
24. A physical medium as in claim 16 , wherein
said instructions to provision and direct include instructions to
provision said resources and direct said resources for a plurality of test suites to be performed substantially concurrently.
25. A physical medium as in claim 16 , wherein
said instructions to provision resources include instructions to
order said instructions to provision in response to a measure of performance of said test suites.
26. A physical medium as in claim 16 , wherein
said instructions to provision resources include instructions to
provision said resources in response to a plurality of test suites, wherein said resources overlap within said plurality of test suites; and
wherein said instructions to provision are performed substantially before directing any said resources to perform operations.