Autonomous organization and role selection of homogenous workers
A method for configuring replicas in a distributed computing system is disclosed. In one example embodiment, a plurality of replicas with associated bootstrap modules may be created. The same bootstrap module code may be used for each replica, thereby simplifying configuration. Using the bootstrap module, the replicas may automatically configure themselves and self-assign a role for a set of predetermined roles such as master and worker. The bootstrap module may check a predetermined location such as a shared network folder for earlier registration entries and then self-select based on the remaining available roles. The bootstrap module may also store its own registration entry to inform subsequent replicas of the role and network address for the current replica so that they may self-configure correctly.
1. A method comprising:
creating a plurality of replicas, and
creating a bootstrap module for each replica of the plurality of replicas such that a plurality of bootstrap modules are created, wherein each bootstrap module of the plurality of bootstrap modules is configured to:
(i) read from a predetermined location to check for earlier-stored registration entries, wherein the predetermined location each bootstrap module reads from is the same;
(ii) in response to reading not more than a predetermined number of earlier-stored registration entries at the predetermined location, configure its replica in a first role and store a respective registration entry at the predetermined location; and
(iii) in response to reading more than the predetermined number of earlier-stored registration entries at the predetermined location, configure its replica in a second role and store the respective registration entry at the predetermined location.
2. The method of claim 1 , wherein the predetermined number of earlier-stored registration entries is zero, the first role is a master role, and the second role is a worker role.
3. The method of claim 2 , wherein each of the plurality of bootstrap modules is the same, regardless of whether the bootstrap module's associated replica is to be configured in the worker role or the master role.
4. The method of claim 1 , wherein each bootstrap module of the plurality of bootstrap modules is further configured to (i) determine how many replicas are to act in the first role and (ii) determine how many replicas are to act in the second role.
5. The method of claim 1 , wherein the predetermined location includes configuration data, wherein the configuration data identifies at least how many first role replicas and second role replicas are to be created.
6. The method of claim 2 , wherein the registration entry comprises metadata.
7. The method of claim 6 , wherein the metadata comprises at least a network address.
8. The method of claim 2 , wherein the predetermined location is a shared network directory.
9. The method of claim 8 , wherein the shared network directory is named based on a per-job globally unique identifier (GUID).
10. The method of claim 2 , further comprising reading a policy from the predetermined location.
11. The method of claim 2 , further comprising, if one or more of the earlier-stored registration entries are present, reading an address of a master replica from the predetermined location.
12. The method of claim 2 , further comprising, if one or more earlier-stored registration entries from other replicas are not present, waiting and reading network addresses of one or more workers from the predetermined location.
13. The method of claim 2 , wherein the bootstrap modules are further configured to:
(v) monitor the predetermined location for additional registration entries from other replicas.
14. The method of claim 13 , further comprising:
monitoring the plurality of replicas, and
in response to detecting non-responsiveness of a critical number of replicas, clearing the registration entries from the predetermined location, and resetting the plurality of replicas.
15. The method of claim 2 , wherein the registration entry is a semaphore.
16. The method of claim 2 , wherein the registration entry is a file stored in a file system.
17. The method of claim 2 , wherein the replicas are for compute or storage.
18. The method of claim 2 , wherein the bootstrap modules are shell scripts configured to run on startup.
19. A method comprising:
creating a plurality of replicas including at least a first replica and a second replica;
creating a bootstrap module for each replica such that a plurality of bootstrap modules is created, wherein each bootstrap module of the plurality of bootstrap modules is configured to:
(i) auto execute on startup;
(ii) check for registration entries from other bootstrap modules of the plurality of bootstrap modules, wherein checking for registration entries comprises reading from a predetermined location for the registration entries, wherein the predetermined location is a shared location that each bootstrap module of the plurality of bootstrap modules checks for the registration entries from the other bootstrap modules;
(iii) in response to reading not more than a predetermined number of registration entries at the predetermined location, (a) create a first replica registration entry for the first replica at the predetermined location and (b) configure the first replica to act in a master role; and
(iv) in response to reading more than the predetermined number of registration entries at the predetermined location, (a) create a second replica registration entry for the second replica at the predetermined location and (b) configure the second replica to act in a worker role.
20. The method of claim 19 , wherein each bootstrap module of the plurality of bootstrap modules is further configured to (i) determine how many replicas are to act in the worker role and (ii) determine how many replicas are to act in at least one other role different than the worker role and the master role.