Fault-tolerant and highly available configuration of distributed services
Fault-tolerant and highly available configuration of distributed services including a computer-implemented method for role-based configuration discovery comprising receiving a request comprising an identifier of a role; identifying a first key, in a replica of a distributed configuration store, comprising a first value that matches the role identifier; identifying one or more other key-value pairs associated in the replica with the first key; and returning a response to an entity that sent the request comprising the value of at least one key-value pair that is specific to the role the service has. Also disclosed are techniques for log forwarding.
1. A method for log forwarding, the method comprising:
obtaining, at a host, network endpoint information from a replica of a distributed configuration store that is stored locally at the host, the replica of the distributed configuration store updated using a consensus protocol that is configured to keep network endpoint information of multiple replicas of the distributed configuration store consistent;
wherein the network endpoint information identifies a location on a network of a first service, the first service comprising a centralized log collection and aggregation network service;
obtaining, at the host, an identifier of a second service installed on the host from the replica of the distributed configuration store, the second service being distinct from the first service;
based, at least in part, on the obtaining the identifier of the second service, obtaining, at the host, from the replica of the distributed configuration store, information indicating where, in a file system, one or more logs generated by the second service are stored;
using the information indicating where the one or more logs generated by the second service are stored, collecting, at the host, the one or more logs generated by the second service;
using the network endpoint information, sending, by the host, the one or more logs generated by the second service to the first service.
2. The method of claim 1 , wherein the network endpoint information comprises an interne protocol address associated with the first service for collecting logs.
3. The method of claim 1 , further comprising:
based, at least in part, on the obtaining the identifier of the second service, providing, to the first service, the identifier of the second service in association with the information from the one or more logs provided to the first service.
4. The method of claim 1 , further comprising:
obtaining a host identifier of the host at which the replica is stored; and
based, at least in part, on the obtaining the host identifier, obtaining, from the replica, information indicating where, in a file system, the one or more logs generated by the second service are stored.
5. The method of claim 4 , wherein the sending the information from the one or more logs to the first service is based, at least in part, on the obtaining the information indicating where, in the file system, the one or more logs are stored.
6. One or more non-transitory computer-readable media storing one or more programs for execution by one or more processors, the one or more programs comprising instructions for:
obtaining, at a host, network endpoint information from a replica of a distributed configuration store that is stored locally at the host, the replica of the distributed configuration store updated using a consensus protocol that is configured to keep network endpoint information of multiple replicas of the distributed configuration store consistent;
wherein the network endpoint information identifies a location on a network of a first service, the first service comprising a centralized log collection and aggregation network service;
obtaining, at the host, an identifier of a second service installed on the host from the replica of the distributed configuration store, the second service being distinct from the first service;
based, at least in part, on the obtaining the identifier of the second service, obtaining, at the host, from the replica of the distributed configuration store, information indicating where, in a file system, one or more logs generated by the second service are stored;
using the information indicating where the one or more logs generated by the second service are stored, collecting, at the host, the one or more logs generated by the second service;
using the network endpoint information, sending, by the host, the one or more logs generated by the second service to the first service.
7. The one or more non-transitory computer-readable media of claim 6 , wherein the network endpoint information comprises an internet protocol address associated with the first service for collecting logs.
8. The one or more non-transitory computer-readable media of claim 6 , further comprising instructions which, when executed by the one or more processors, cause:
based, at least in part, on the obtaining the identifier of the second service, providing, to the first service, the identifier of the second service in association with the information from the one or more logs provided to the first service.
9. The one or more non-transitory computer-readable media of claim 6 , further comprising instructions which, when executed by the one or more processors, cause:
obtaining a host identifier of the host at which the replica is stored; and
based, at least in part, on the obtaining the host identifier, obtaining, from the replica, information indicating where, in a file system, the one or more logs generated by the second service are stored.
10. The one or more non-transitory computer-readable media of claim 9 , wherein the sending the information from the one or more logs to the first service is based, at least in part, on the obtaining the information indicating where, in the file system, the one or more logs are stored.
11. A system, comprising:
one or more processors;
one or more storage media storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for:
obtaining, at a host, network endpoint information from a replica of a distributed configuration store that is stored locally at the host, the replica of the distributed configuration store updated using a consensus protocol that is configured to keep network endpoint information of multiple replicas of the distributed configuration store consistent;
wherein the network endpoint information identifies a location on a network of a first service, the first service comprising a centralized log collection and aggregation network service;
obtaining, at the host, an identifier of a second service installed on the host from the replica of the distributed configuration store, the second service being distinct from the first service;
based, at least in part, on the obtaining the identifier of the second service, obtaining, at the host, from the replica of the distributed configuration store, information indicating where, in a file system, one or more logs generated by the second service are stored;
using the information indicating where the one or more logs generated by the second service are stored, collecting, at the host, the one or more logs generated by the second service;
using the network endpoint information, sending, by the host, the one or more logs generated by the second service to the first service.
12. The system of claim 11 , wherein the network endpoint information comprises an interne protocol address associated with the first service for collecting logs.
13. The system of claim 11 , further comprising instructions which, when executed by the one or more processors, cause:
based, at least in part, on the obtaining the identifier of the second service, providing, to the first service, the identifier of the second service in association with the information from the one or more logs provided to the first service.
14. The system of claim 11 , further comprising instructions which, when executed by the one or more processors, cause:
obtaining a host identifier of the host at which the replica is stored; and
based, at least in part, on the obtaining the host identifier, obtaining, from the replica, information indicating where, in a file system, the one or more logs generated by the second service are stored.
15. The system of claim 14 , wherein the sending the information from the one or more logs to the first service is based, at least in part, on the obtaining the information indicating where, in the file system, the one or more logs are stored.