Two-way replication of search node configuration files using a mediator node
In a computer-implemented method for configuring a distributed computer system comprising a plurality of nodes of a plurality of node classes, configuration files for a plurality of nodes of each of the plurality of node classes are stored in a central repository. The configuration files include information representing a desired system state of the distributed computer system, and the distributed computer system operates to keep an actual system state of the distributed computer system consistent with the desired system state. The plurality of node classes includes forwarder nodes for receiving data from an input source, indexer nodes for indexing the data, and search head nodes for searching the data. Responsive to receiving changes to the configuration files, the changes are propagated to nodes of the plurality of nodes impacted by the changes based on a node class of the nodes impacted by the changes.
1. A computer-implemented method comprising:
storing, in a central repository on a management node, configuration files for a plurality of node classes of a data intake and query system; and
propagating changes, to one of the configuration files for one of the node classes comprising search nodes responsible for performing search operations on data collected by the data intake and query system, using two-way replication between the management node and the search nodes via a mediator node of the search nodes.
2. The computer-implemented method of claim 1 , further comprising propagating other changes, to another one of the configuration files for another one of the node classes, using one-way replication from the management node to nodes of the other node class via another mediator node of the other node class.
3. The computer-implemented method of claim 1 , wherein the mediator node is in a cluster of the search nodes, wherein the mediator node includes a search head configuration repository that manages, for each of the search nodes in the cluster, configuration checkpoints and deltas from the configuration checkpoints.
4. The computer-implemented method of claim 1 , wherein the search nodes are configured to asynchronously replicate local configuration files to the mediator node by transmitting deltas from configuration checkpoints.
5. The computer-implemented method of claim 1 , wherein the search nodes are configured to asynchronously replicate local configuration files to the mediator node by pulling a configuration checkpoint from the mediator node, merging local changes to the local configuration files with the configuration checkpoint to generate merged deltas, and pushing the merged deltas to the mediator node.
6. The computer-implemented method of claim 1 , wherein using the two-way replication comprises the search nodes pulling a configuration checkpoint from the mediator node, overwriting a local configuration file using the configuration checkpoint, and subsequently pushing to the mediator node and pulling from the mediator node deltas from the configuration checkpoint.
7. The computer-implemented method of claim 1 , wherein the mediator node is in a cluster of the search nodes, the computer-implemented method further comprising, upon partitioning the cluster into a majority partition and a minority partition, buffering deltas from configuration checkpoints for a set of the search nodes in the minority partition until the minority partition is healed.
8. The computer-implemented method of claim 1 , wherein the mediator node is in a cluster of the search nodes, wherein the search nodes are configured to pull snapshots of a search head configuration repository from the mediator node, the computer-implemented method further comprising, upon partitioning the cluster into a majority partition and a minority partition, configuring one of the search nodes in the minority partition as a new mediator node for the minority partition using one of the snapshots of the search head configuration repository for the new mediator node.
9. One or more non-transitory computer storage media containing instructions which when executed on one or more data processors, cause the one or more data processors to perform operations comprising:
storing, in a central repository on a management node, configuration files for a plurality of node classes of a data intake and query system; and
propagating changes, to one of the configuration files for one of the node classes comprising search nodes responsible for performing search operations on data collected by the data intake and query system, using two-way replication between the management node and the search nodes via a mediator node of the search nodes.
10. The one or more non-transitory computer storage media of claim 9 , the operations further comprising propagating other changes, to another one of the configuration files for another one of the node classes, using one-way replication from the management node to nodes of the other node class via another mediator node of the other node class.
11. The one or more non-transitory computer storage media of claim 9 , wherein the mediator node is in a cluster of the search nodes, wherein the mediator node include s a search head configuration repository configured to manage, for each of the search nodes in the cluster, configuration checkpoints and deltas from the configuration checkpoints.
12. The one or more non-transitory computer storage media of claim 9 , wherein the search nodes are configured to asynchronously replicate local configuration files to the mediator node by transmitting deltas from configuration checkpoints.
13. The one or more non-transitory computer storage media of claim 9 , wherein the search nodes are configured to asynchronously replicate local configuration files to the mediator node by pulling a configuration checkpoint from the mediator node, merging local changes to the local configuration files with the configuration checkpoint to generate merged deltas, and pushing the merged deltas to the mediator node.
14. The one or more non-transitory computer storage media of claim 9 , wherein using the two-way replication comprises the search nodes pulling a configuration checkpoint from the mediator node, overwriting a local configuration file using the configuration checkpoint, and subsequently pushing to the mediator node and pulling from the mediator node deltas from the configuration checkpoint.
15. The one or more non-transitory computer storage media of claim 9 , wherein the mediator node is in a cluster of the search nodes, the operations further comprising, upon partitioning the cluster into a majority partition and a minority partition, buffering deltas from configuration checkpoints for a set of the search nodes in the minority partition until the minority partition is healed.
16. The one or more non-transitory computer storage media of claim 9 , wherein the mediator node is in a cluster of the search nodes, wherein the search nodes are configured to pull snapshots of a search head configuration repository from the mediator node, the operations further comprising, upon partitioning the cluster into a majority partition and a minority partition, configuring one of the search nodes in the minority partition as a new mediator node for the minority partition using one of the snapshots of the search head configuration repository for the new mediator node.
17. A system comprising:
one or more data processors; and
one or more non-transitory computer storage media containing instructions which when executed on the one or more data processors, cause the one or more processors to perform operations comprising:
storing, in a central repository on a management node, configuration files for a plurality of node classes of a data intake and query system; and
propagating changes, to one of the configuration files for one of the node classes comprising search nodes responsible for performing search operations on data collected by the data intake and query system, using two-way replication between the management node and the search nodes via a mediator node of the search nodes.
18. The system of claim 17 , wherein the search nodes are configured to asynchronously replicate local configuration files to the mediator node by transmitting deltas from configuration checkpoints.
19. The system of claim 17 , wherein the search nodes are configured to asynchronously replicate local configuration files to the mediator node by pulling a configuration checkpoint from the mediator node, merging local changes to the local configuration files with the configuration checkpoint to generate merged deltas, and pushing the merged deltas to the mediator node.
20. The system of claim 17 , wherein using the two-way replication comprises the search nodes pulling a configuration checkpoint from the mediator node, overwriting a local configuration file using the configuration checkpoint, and subsequently pushing to the mediator node and pulling from the mediator node deltas from the configuration checkpoint.