Data replication and site replication in a clustered computing environment
A method of data replication in a clustered computing environment comprises receiving, at a selected indexer within a plurality of indexers in a cluster, data from a forwarder indexer, wherein the selected indexer is designated as a primary indexer for the data, wherein the primary indexer has primary responsibility for responding to search queries pertaining to the data, wherein the cluster comprises a plurality of sites. The method further comprises receiving, at the selected indexer, data replication instructions, wherein the data replication instructions comprise a number of other indexers in the cluster for storing a replicated copy of the data and further comprise a number of sites from the plurality of sites across which to store a replicated copy of the data determined in accordance with a site replication factor.
1. A method, comprising:
receiving, at a selected indexer within a plurality of indexers in a cluster, data from a forwarder indexer, wherein the selected indexer is designated as a primary indexer for the data, wherein the primary indexer is operable to respond to search queries pertaining to the data, wherein the cluster comprises a plurality of sites, wherein each site of the plurality of sites comprises a subset of the plurality of indexers;
receiving, at the selected indexer, data replication instructions from a master node of the cluster, wherein the data replication instructions identify a number of other indexers in the cluster for storing a replicated copy of the data determined in accordance with a replication factor configured for the cluster, and wherein the data replication instructions further identify a number of sites from the plurality of sites across which to store the replicated copy of the data determined in accordance with a site replication factor; and
transmitting, from the selected indexer, the data to the other indexers for replication.
2. The method of claim 1 , wherein the replication factor and the site replication factor are included in the data replication instructions.
3. The method of claim 1 , wherein each site of the plurality of sites is associated with a separate geographical location.
4. The method of claim 1 , wherein, for each of the other indexers, the data replication instructions indicate whether a respective indexer is to store a searchable copy of the data.
5. The method of claim 1 , wherein each site of the plurality of sites is associated with a separate geographical location, and wherein at least one of the other indexers is located at a different site from the plurality of sites than the selected indexer.
6. The method of claim 1 , wherein, for each of the other indexers, the data replication instructions indicate whether a respective indexer is to store a searchable copy of the data, and wherein further a number of searchable copies to be replicated on the other indexers is determined in accordance with a search factor configured for the cluster.
7. The method of claim 1 , wherein, for each of the other indexers, the data replication instructions indicate whether a respective indexer is to store a searchable copy of the data, and wherein further the searchable copy of the data comprises index files, wherein the index files are operable to be searched in response to search requests.
8. The method of claim 1 , wherein, for each of the other indexers, the data replication instructions indicate whether a respective indexer is to store a searchable copy of the data, and wherein the respective indexer generates the searchable copy of the data by processing the data to generate a separate keyword index.
9. The method of claim 1 , wherein the forwarder indexer is one of a plurality of forwarder indexers, each forwarder indexer operating independent of each other forwarder indexer to select an indexer as a primary indexer for data sent to the cluster by a respective forwarder indexer of the plurality of indexers.
10. The method of claim 1 , wherein the site replication factor is based on one of a type of data to be replicated and information from a forwarder indexer.
11. The method of claim 1 , wherein at least one indexer in the plurality of indexers is designated as a secondary indexer for the data and as a primary indexer for different data.
12. The method of claim 1 , wherein the data replication instructions are generated by the master node in accordance with a data replication policy for the cluster.
13. The method of claim 1 , wherein the other indexers are determined by the master node of the cluster based on load balancing criteria.
14. The method of claim 1 , wherein the selected indexer is designated as a primary indexer for the data at a specified generation identifier, and wherein the other indexers are each designated as secondary indexers for the data at the specified generation identifier.
15. The method of claim 1 , wherein the selected indexer is designated as a primary indexer for the data at a specified generation identifier, wherein the other indexers are each designated as secondary indexers for the data at the specified generation identifier, and further wherein for a given query request for the data at the specified generation identifier received by each of the indexers in the cluster, the primary indexer for the data at the specified generation identifier responds to the query request, and each secondary indexer for the data at the specified generation identifier ignores the request.
16. The method of claim 1 , wherein the selected indexer is designated as a primary indexer for the data at a specified generation identifier, and wherein the other indexers are each designated as secondary indexers for the data at the specified generation identifier, and wherein further the master node of the cluster is configured to assign the specified generation identifier to the data, and further is configured to select at least one of the secondary indexers to become a new primary indexer of the data when the primary indexer for the specified generation identifier is determined to be non-responsive.
17. The method of claim 1 , wherein the transmitting the data to the other indexers for replication comprises transmitting the data with a journal comprising metadata useable to recreate the data.
18. The method of claim 1 , wherein each indexer of the plurality of indexers stores search affinity information, which indicates, for each subset of data stored by a respective indexer and for each site from which a query originates, whether the respective indexer has primary responsibility for returning search results for a respective subset of data.
19. A non-transitory computer-readable medium storing computer-executable instructions which, when executed by a processor, cause the processor to perform operations comprising:
receiving, at a selected indexer within a plurality of indexers in a cluster, data from a forwarder indexer, wherein the selected indexer is designated as a primary indexer for the data, wherein the primary indexer is operable to respond to search queries pertaining to the data, wherein the cluster comprises a plurality of sites, wherein each site of the plurality of sites comprises a subset of the plurality of indexers;
receiving, at the selected indexer, data replication instructions from a master node of the cluster, wherein the data replication instructions identify a number of other indexers in the cluster for storing a replicated copy of the data determined in accordance with a replication factor configured for the cluster, and wherein the data replication instructions further identify a number of sites from the plurality of sites across which to store the replicated copy of the data determined in accordance with a site replication factor; and
transmitting, from the selected indexer, the data to the other indexers for replication.
20. A system comprising:
at least one memory storing computer-executable instructions; and
at least one processor, wherein the at least one processor is configured to access the at least one memory and to execute the computer-executable instructions to:
receive, at a selected indexer within a plurality of indexers in a cluster, data from a forwarder indexer, wherein the selected indexer is designated as a primary indexer for the data, wherein the primary indexer is operable to respond to search queries pertaining to the data, wherein the cluster comprises a plurality of sites, wherein each site of the plurality of sites comprises a subset of the plurality of indexers;
receive, at the selected indexer, data replication instructions from a master node of the cluster, wherein the data replication instructions identifies a number of other indexers in the cluster for storing a replicated copy of the data determined in accordance with a replication factor configured for the cluster, and wherein the data replication instructions further identifies a number of sites from the plurality of sites across which to store the replicated copy of the data determined in accordance with a site replication factor; and
transmit, from the selected indexer, the data to the other indexers for replication.