Systems and methods for updating a status indication in a system providing dynamic indexer discovery
The present invention is related to a method for providing dynamic indexer discovery. The method comprises receiving, from an index manager, a status indication associated with a plurality of indexers, wherein each of the plurality of indexers indexes events of raw machine-generated data received from a plurality of data collectors. The method further comprises determining a weight associated with each of the plurality of indexers and selecting an indexer from the plurality of indexers. Subsequently, the method comprises allocating data to the indexer in accordance with a respective weight assigned to the indexer and transmitting the allocated data to the indexer.
1 . A computer-implemented method comprising:
receiving, via a data collector that obtains information from one or more sources and forwards the information to one or more indexers for processing, a performance metric from each indexer included in a set of indexers;
assigning, via the data collector, a weight to each indexer based on the performance metric;
receiving, via the data collector, data to be indexed by the set of indexers;
selecting, via the data collector, a first indexer from the set of indexers to process the data to be indexed based on the performance metrics; and
forwarding, via the data collector, the data to the first indexer for processing to balance load across the one or more indexers.
2 . The computer-implemented method of claim 1 , wherein assigning the weight to each indexer comprises determining a storage capacity of each indexer based upon the performance metric for each indexer, wherein the weight is based upon the storage capacity.
3 . The computer-implemented method of claim 1 , wherein assigning the weight to each indexer comprises:
determining a total storage capacity of the set of indexers;
determining a storage capacity of each indexer included in the set of indexers based on the performance metric for each indexer; and
assigning the weight to each indexer based on the determined storage capacity of each indexer included in the set of indexers and the total storage capacity.
4 . The computer-implemented method of claim 1 , wherein the performance metric for each indexer is based on a processing capacity or a network connectivity associated with each indexer included in the set of indexers.
5 . The computer-implemented method of claim 1 , wherein selecting the first indexer comprises selecting an indexer from the set of indexers with a greatest amount of indexing capacity.
6 . The computer-implemented method of claim 1 , wherein selecting the first indexer comprises pseudo-randomly selecting an indexer from the set of indexers based on the performance metric.
7 . The computer-implemented method of claim 1 , wherein selecting the first indexer comprises selecting an indexer from the set of indexers based on the performance metric for each indexer, wherein the first indexer with a first performance metric indicating a higher performance is selected more often than a second indexer with a second performance metric indicating a lower performance.
8 . The computer-implemented method of claim 1 , wherein selecting the first indexer comprises selecting the first indexer in proportion to the performance metric of the first indexer relative to the performance metric of other indexers from the set of indexers.
9 . The computer-implemented method of claim 1 , wherein the first indexer indexes the data for a subsequent searching operation.
10 . The computer-implemented method of claim 1 , further comprising prior to forwarding the data to the first indexer, collecting the data from a plurality of data collectors in a data center environment.
11 . A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
receiving, via a data collector that obtains information from one or more sources and forwards the information to one or more indexers for processing, a performance metric from each indexer included in a set of indexers;
assigning, via the data collector, a weight to each indexer based on the performance metric;
receiving, via the data collector, data to be indexed by the set of indexers;
selecting, via the data collector, a first indexer from the set of indexers to process the data to be indexed based on the performance metrics; and
forwarding, via the data collector, the data to the first indexer for processing to balance load across the one or more indexers.
12 . The computing device of claim 11 , wherein the data corresponds to raw machine-data obtained from a plurality of data collectors.
13 . The computing device of claim 11 , wherein selecting the first indexer comprises selecting an indexer from the set of indexers based on an amount of excess capacity indicated by the performance metric for each indexer.
14 . The computing device of claim 11 , wherein assigning the weight to each indexer comprises determining a storage capacity of each indexer based upon on the performance metric for each indexer, wherein the weight is based upon the storage capacity.
15 . The computing device of claim 11 , wherein assigning the weight to each indexer comprises:
determining a total storage capacity of the set of indexers;
determining a storage capacity of each indexer included in the set of indexers based on the performance metric for each indexer; and
assigning the weight to each indexer based on the determined storage capacity of each indexer included in the set of indexers and the total storage capacity.
16 . The computing device of claim 11 , wherein the performance metric for each indexer is based on a processing capacity or a network connectivity associated with each indexer included in the set of indexers.
17 . The computing device of claim 11 , wherein selecting the first indexer comprises pseudo-randomly selecting an indexer from the set of indexers based on the performance metric.
18 . The computing device of claim 11 , wherein selecting the first indexer comprises selecting the first indexer in proportion to the performance metric of the first indexer relative to the performance metric of other indexers from the set of indexers.
19 . The computing device of claim 11 , wherein the first indexer indexes the data for a subsequent searching operation.
20 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
receiving, via a data collector that obtains information from one or more sources and forwards the information to one or more indexers for processing, a performance metric from each indexer included in a set of indexers;
assigning, via the data collector, a weight to each indexer based on the performance metric;
receiving, via the data collector, data to be indexed by the set of indexers;
selecting, via the data collector, a first indexer from the set of indexers to process the data to be indexed based on the performance metrics; and
forwarding, via the data collector, the data to the first indexer for processing to balance load across the one or more indexers.