Implementing a split-brain prevention strategy when configuring automatic cluster manager failover

1. A computer-implemented method of managing dynamic cluster manager failover, the method comprising:

routing data traffic associated with managing a plurality of indexers in a cluster to a first cluster manager, wherein the first cluster manager is associated with an active role and is operable to manage the plurality of indexers in the cluster in a capacity of the active role;

transmitting periodic heartbeat request messages from a second cluster manager of the cluster to the first cluster manager, wherein the second cluster manager is associated with a standby role;

detecting, at the second cluster manager, a loss of heartbeat response messages from the first cluster manager, wherein the heartbeat response messages are transmitted from the first cluster manager to the second cluster manager in response to the periodic heartbeat request message and are operable to communicate a status of the first cluster manager to the second cluster manager;

selecting, at the second cluster manager, a set of indexers from the plurality of indexers to establish communication with the first cluster manager;

receiving information regarding the first cluster manager from the set of indexers, wherein the information indicates a status of the first cluster manager; and

in response to a determination that the status of the first cluster manager is offline, promoting the second cluster manager to switch over to the active role, otherwise, maintaining the first cluster manage in the active role.

2. The method of claim 1 , wherein the promoting comprises:

re-routing the data traffic associated with managing the plurality of indexers to the second cluster manager.

3. The method of claim 1 , wherein the detecting comprises determining that the loss of the heartbeat response messages has exceeded a predetermined duration of time.

4. The method of claim 1 , wherein the detecting comprises determining that a predetermined number of heartbeat request messages did not receive corresponding heartbeat response messages from the second cluster manager.

5. The method of claim 1 , wherein the selecting the set of indexers comprises non-deterministically choosing between active indexers indicated on a status map received from the first cluster manager.

6. The method of claim 1 , wherein the plurality of indexers are dispersed over multiple sites, and wherein the set of indexers comprises one or more indexers from each of the multiple sites.

7. The method of claim 1 , wherein the determination that the status of the first cluster manager is online is based on the information indicating that at least one indexer from the set of indexers recently established communication with the first cluster manager.

8. The method of claim 1 , wherein the promoting comprises re-routing the data traffic associated with managing the plurality of indexers to the second cluster manager, and wherein the determination that the status of the first cluster manager is offline is based on the information indicating at least half the indexers from the set of indexers voted that the first cluster manager is offline.

9. The method of claim 1 , further comprising: wherein the determining further comprises:

responsive to a determination that the set of indexers has not achieved consensus regarding the status of the first cluster manager, retrying the selecting and the receiving using exponentially backed off intervals, wherein a non-deterministic selection of indexers from the set of indexers is used at each exponentially backed off interval.

10. The method of claim 1 , further comprising:

responsive to a determination that the set of indexers cannot be contacted, maintaining the first cluster manager in the active role.

11. The method of claim 1 , wherein the routing is performed using a proxy server or a load balancer.

12. A computing device, comprising:

a processor; and

a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:

routing data traffic associated with managing a plurality of indexers in a cluster to a first cluster manager, wherein the first cluster manager is associated with an active role and is operable to manage the plurality of indexers in the cluster in a capacity of the active role;

transmitting periodic heartbeat request messages from a second cluster manager of the cluster to the first cluster manager, wherein the second cluster manager is associated with a standby role;

detecting, at the second cluster manager, a loss of heartbeat response messages from the first cluster manager, wherein the heartbeat response messages are transmitted from the first cluster manager to the second cluster manager in response to the periodic heartbeat request message and are operable to communicate a status of the first cluster manager to the second cluster manager;

selecting, at the second cluster manager, a set of indexers from the plurality of indexers to establish communication with the first cluster manager;

receiving information regarding the first cluster manager from the set of indexers, wherein the information indicates a status of the first cluster manager; and

in response to a determination that the status of the first cluster manager is offline, promoting the second cluster manager to switch over to the active role, otherwise, maintaining the first cluster manage in the active role.

13. The computing device of claim 12 , wherein the promoting comprises:

re-routing data traffic associated with the managing the plurality of indexers to the second cluster manager.

14. The computing device of claim 12 , wherein the selecting the set of indexers comprises non-deterministically choosing between active indexers indicated on a status map received from the first cluster manager.

15. The computing device of claim 12 , wherein the plurality of indexers are dispersed over multiple sites, and wherein the set of indexers comprises one or more indexers from each of the multiple sites.

16. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:

routing data traffic associated with managing a plurality of indexers in a cluster to a first cluster manager, wherein the first cluster manager is associated with an active role and is operable to manage the plurality of indexers in the cluster in a capacity of the active role;

transmitting periodic heartbeat request messages from a second cluster manager of the cluster to the first cluster manager, wherein the second cluster manager is associated with a standby role;

detecting, at the second cluster manager, a loss of heartbeat response messages from the first cluster manager, wherein the heartbeat response messages are transmitted from the first cluster manager to the second cluster manager in response to the periodic heartbeat request message and are operable to communicate a status of the first cluster manager to the second cluster manager;

selecting, at the second cluster manager, a set of indexers from the plurality of indexers to establish communication with the first cluster manager;

receiving information regarding the first cluster manager from the set of indexers, wherein the information indicates a status of the first cluster manager; and

in response to a determination that the status of the first cluster manager is offline, promoting the second cluster manager to switch over to the active role, otherwise, maintaining the first cluster manage in the active role.

17. The non-transitory computer-readable medium of claim 16 , wherein the promoting comprises:

re-routing data traffic associated with the managing the plurality of indexers to the second cluster manager.

18. The non-transitory computer-readable medium of claim 16 , wherein the promoting comprises re-routing the data traffic associated with managing the plurality of indexers to the second cluster manager, and wherein the determination that the status of the first cluster manager is offline is based on the information indicating at least half the indexers from the set of indexers voted that the first cluster manager is offline.

19. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:

responsive to a determination that the set of indexers has not achieved consensus regarding the status of the first cluster manager, retrying the selecting and the receiving using exponentially backed off intervals, wherein a non-deterministic selection of indexers from the set of indexers is used at each exponentially backed off interval.

20. The non-transitory computer-readable medium of claim 16 , wherein the plurality of indexers are dispersed over multiple sites, and wherein the set of indexers comprises one or more indexers from each of the multiple sites.