Fault-tolerant and highly available configuration of distributed services
Fault-tolerant and highly available configuration of distributed services including a computer-implemented method for role-based configuration discovery comprising receiving a request comprising an identifier of a role; identifying a first key, in a replica of a distributed configuration store, comprising a first value that matches the role identifier; identifying one or more other key-value pairs associated in the replica with the first key; and returning a response to an entity that sent the request comprising the value of at least one key-value pair that is specific to the role the service has. Also disclosed are techniques for log forwarding.
1. A method for service lifecycle management, the method comprising:
obtaining, at a first host of one or more hosts in a distributed computing environment, service-host binding information from a replica of a distributed configuration store, wherein the replica of the distributed configuration store represents a local copy of the service-host binding information;
determining, at the first host, one or more services to watch based on the service-host binding information, a service of the one or more services being a single instance of a software product or software application installed on at least the first host or one of the one or more hosts in a distributed computing environment;
setting a watch on the one or more services to watch;
based on the watch, receiving a notification that a target state of a particular service of the one or more services to watch is modified, comprising updating target state information in the replica of the distributed configuration store using a consensus protocol that allows hosts to agree on an ordering for changes in the distributed configuration store;
wherein the updated target state of the particular service represents a desired operational state for the particular service;
determining, at the first host, a current state of the particular service of the one or more services to watch, wherein the current state represents an operational state for the particular service, and wherein the particular service is hosted at the first host;
determining whether the current state of the particular service is inconsistent with the updated target state of the particular service of the one or more services to watch;
in response to determining that the current state of the particular service is inconsistent with the updated target state of the particular service, the first host taking a corrective action to modify the particular service, which causes the current state of the particular service to be consistent with the updated target state of the particular service.
2. The method of claim 1 , wherein the one or more services to watch is determined based on an identifier of the first host.
3. The method of claim 1 , wherein the one or more services to watch is determined by enumerating one or more child keys of a predefined key that includes an identifier of the first host in a key name associated with the predefined key.
4. The method of claim 1 , wherein the watch comprises at least one of: a forward watch, a historical and forward watch, or a non-recursive watch.
5. The method of claim 1 , wherein the corrective action taken to modify the particular service by the first host is based on the updated target state of the particular service.
6. The method of claim 1 , further comprising:
in response to determining that the current state of the particular service at the first host is not inconsistent with the updated target state for the particular service, continuing to watch the one or more services to watch.
7. One or more non-transitory computer-readable media storing one or more programs for execution by one or more processors, the one or more programs comprising instructions for:
obtaining, at a first host, service-host binding information from a replica of a distributed configuration store;
determining, at the first host, one or more services to watch based on the service-host binding information, a service of the one or more services being a single instance of a software product or software application installed on one or more hosts in a distributed computing environment;
setting a watch on the one or more services to watch;
receiving a notification that a target state of a particular service of the one or more services to watch is modified, comprising updating target state information in the replica of the distributed configuration store using a consensus protocol that allows hosts to agree on an ordering for changes in the distributed configuration store;
wherein the target state of the particular service represents a desired operational state for the particular service;
determining, at the first host, a current state of the particular service of the one or more services to watch, wherein the current state represents an operational state for the particular service, and wherein the particular service is hosted at the first host;
determining whether the current state of the particular service is inconsistent with the target state of the particular service of the one or more services to watch;
in response to determining that the current state of the particular service is inconsistent with the target state of the particular service, the first host taking a corrective action to modify the particular service, which causes the current state of the particular service to be consistent with the target state of the particular service.
8. The one or more non-transitory computer-readable media of claim 7 , wherein the one or more services to watch is determined based on an identifier of the first host.
9. The one or more non-transitory computer-readable media of claim 7 , wherein the one or more services to watch is determined by enumerating one or more child keys of a predefined key that includes an identifier of the first host in a key name associated with the predefined key.
10. The one or more non-transitory computer-readable media of claim 7 , wherein the watch comprises at least one of: a forward watch, a historical and forward watch, or a non-recursive watch.
11. The one or more non-transitory computer-readable media of claim 7 , wherein the corrective action taken to modify the particular service by the first host is based on the updated target state of the particular service.
12. The one or more non-transitory computer-readable media of claim 7 , further comprising instructions which, when executed by the one or more processors, cause:
in response to determining that the current state of the particular service at the first host is not inconsistent with the updated target state for the particular service, continuing to watch the one or more services to watch.
13. A system, comprising:
one or more processors;
one or more storage media storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for:
obtaining, at a first host of one or more hosts in a distributed computing environment, service-host binding information from a replica of a distributed configuration store, wherein the replica of the distributed configuration store represents a local copy of the service-host binding information;
determining, at the first host, one or more services to watch based on the service-host binding information, a service of the one or more services being a single instance of a software product or software application installed on at least the first host or one of the one or more hosts in a distributed computing environment;
setting a watch on the one or more services to watch;
based on the watch, receiving a notification that a target state of a particular service of the one or more services to watch is modified, comprising updating target state information in the replica of the distributed configuration store using a consensus protocol that allows hosts to agree on an ordering for changes in the distributed configuration store;
wherein the updated target state of the particular service represents a desired operational state for the particular service;
determining, at the first host, a current state of the particular service of the one or more services to watch, wherein the current state represents an operational state for the particular service, and wherein the particular service is hosted at the first host;
determining whether the current state of the particular service is inconsistent with the updated target state of the particular service of the one or more services to watch;
in response to determining that the current state of the particular service is inconsistent with the updated target state of the particular service, the first host taking a corrective action to modify the particular service, which causes the current state of the particular service to be consistent with the updated target state of the particular service.
14. The system of claim 13 , wherein the one or more services to watch is determined based on an identifier of the first host.
15. The system of claim 13 , wherein the one or more services to watch is determined by enumerating one or more child keys of a predefined key that includes an identifier of the first host in a key name associated with the predefined key.
16. The system of claim 13 , wherein the watch comprises at least one of: a forward watch, a historical and forward watch, or a non-recursive watch.
17. The system of claim 13 , wherein the corrective action taken to modify the particular service by the first host is based on the updated target state of the particular service.
18. The system of claim 13 , further comprising instructions which, when executed by the one or more processors, cause:
in response to determining that the current state of the particular service at the first host is not inconsistent with the updated target state for the particular service, continuing to watch the one or more services to watch.