Distributed dynamic system configuration
View Patent ↗Reliably making configuration changes to distributed systems, including receiving commands for multiple configuration changes, subdividing configuration changes into separate tasks, and performing those tasks at each node. A configuration element receives sets of configuration change commands, acknowledging them so the user need not wait before issuing additional commands. Tasks are determined, each including consistent changes to system configuration, and each including single-device tasklets. Each particular tasklet might be assigned to a particular single device, or to any single device in the system. Next tasks are performed when tasklets are complete. If tasklets are not timely performed due to nodes which are relatively unresponsive, those nodes are marked “failed.” When a failed node returns to responsiveness, it marks itself “recovering.” When a recovering node catches up, it marks itself “operational.” Updates by failed or recovering nodes are skipped while synchronizing with operational nodes.
1. A method, including steps of
receiving a plurality of sets of commands, each set indicating one or more configuration changes to a distributed system including a plurality of nodes, each said configuration change including modification of at least one of: configuration information associated with said distributed system, configuration information associated with at least one of said plurality of nodes;
queueing said sets of commands, whereby said sets of commands can be performed;
making said configuration changes in response to each said set of commands, said steps of making being performed asynchronously with respect to said steps of queueing, said steps of making being performed in response to said steps of queueing,
whereby
said configuration changes are made in an order responsive to said steps of receiving;
said sets of commands are received whether or not said configuration changes are completed.
2. A method as in claim 1 , wherein
said steps of receiving include steps of
acknowledging receipt of said sets of commands;
readying receipt for additional said sets of commands.
3. A method as in claim 1 , wherein
said steps of making include steps of
indicating substantial completion of said configuration changes;
indicating whether any further said configuration changes are to be made.
4. A method as in claim 1 , including steps of
determining an active configuration of said distributed system, wherein said steps of making are performed with respect to said active configuration;
identifying a new configuration in response to at least partial results of said steps of making with respect to said active configuration;
determining substantial completion of steps of making with respect to said active configuration;
identifying said new configuration as a new said active configuration in response to said steps of determining substantial completion.
5. A physical medium, including information interpretable by a computing device as instructions, said physical medium being non-transitory,
said instructions directing said computing device to
receive a plurality of sets of commands, each set indicating one or more configuration changes to a distributed system including a plurality of nodes, each said configuration change including modification of at least one of: configuration information associated with said distributed system, configuration information associated with at least one of said plurality of nodes;
queue said sets of commands, whereby said set of commands can be performed;
make said configuration changes in response to each said set of commands, said instructions to make being performed asynchronously with respect to said instructions to queue, said instructions to make being performed in response to said instructions to queue;
whereby
said configuration changes are made in an order responsive to said receiving;
said sets of commands are received whether or not said configuration changes are completed.
6. A physical medium as in claim 5 , wherein
said instructions to receive include
instructions to acknowledge receipt of said sets of commands;
instructions to ready receipt for additional said sets of commands.
7. A physical medium as in claim 5 , wherein said instructions to make include
instructions to indicate substantial completion of said configuration changes;
instructions to indicate whether any further said configuration changes are to be made.
8. A physical medium as in claim 5 , including
instructions to determine an active configuration of said distributed system, wherein said instructions to make are performed with respect to said active configuration;
instructions to identify a new configuration in response to at least partial results of said instructions to make with respect to said active configuration;
instructions to determine substantial completion of said instructions to make with respect to said active configuration;
instructions to identify said new configuration as a new said active configuration in response to said instructions to determine substantial completion.