Health check for primary failover
A primary database and one or more secondary databases are managed by a redundancy manager, e.g., PATRONI, that manages failover to one of the secondary databases upon failure of the primary database. A separate orchestrator monitors status of the host of the primary database and monitors values such as loading, latency, temperature, and/or trends in these values. Upon detecting that the values indicate a risk of failure of the host, the orchestrator preemptively instructs the redundancy manager to perform failover to one of the secondary databases.
1 . An apparatus comprising:
a plurality of computing devices each comprising one or more processing devices and one or more memory devices operably coupled to the one or more processing devices, the plurality of computing devices being configured to:
implement a plurality of databases on a plurality of host computing devices of the plurality of computing devices;
implement a redundancy manager configured to select a primary database from the plurality of databases with a remainder of the plurality of databases being secondary databases and manage failover upon failure of the primary database; and
monitor the primary database independently from the redundancy manager and instruct the redundancy manager to preemptively failover to one of the secondary databases prior to failure of the primary database when a risk of failure is detected.
2 . The apparatus of claim 1 , wherein the plurality of computing devices are configured to monitor the primary database by monitoring loading of a host computing device of the plurality of host computing devices implementing the primary database.
3 . The apparatus of claim 1 , wherein the plurality of computing devices are configured to monitor the primary database by monitoring latency of a software component implementing the primary database.
4 . The apparatus of claim 1 , wherein the plurality of computing devices are configured to monitor the primary database by monitoring a temperature of one or more hardware components of a host computing device of the plurality of host computing devices implementing the primary database.
5 . The apparatus of claim 4 , wherein the one or more hardware components include the one or more processing devices of the host computing device of the plurality of host computing devices implementing the primary database.
6 . The apparatus of claim 4 , wherein the one or more hardware components include one or more storage devices of the host computing device of the plurality of host computing devices implementing the primary database.
7 . The apparatus of claim 1 , wherein the plurality of computing devices are configured to monitor trends in measured values for a host computing device of the plurality of host computing devices implementing the primary database.
8 . The apparatus of claim 1 , wherein the plurality of computing devices are configured to select the one of the secondary databases as having a least risk of failure of the secondary databases.
9 . The apparatus of claim 1 , wherein the redundancy manager is an instance of PATRONI.
10 . The apparatus of claim 9 , wherein the plurality of databases are POSTGRESQL databases.
11 . A method comprising:
implementing a plurality of databases on at least a portion of a plurality of computing devices;
implementing, on the plurality of computing devices, a redundancy manager configured to select a primary database from the plurality of databases with a remainder of the plurality of databases being secondary databases and manage failover upon failure of the primary database;
monitoring, by an orchestrator executing on the plurality of computing devices, the primary database independently from the redundancy manager;
detecting, by the orchestrator, risk of failure of the primary database based on the monitoring of the primary database; and
in response to detecting the risk of failure, instructing, by the orchestrator, the redundancy manager to preemptively failover to one of the secondary databases prior to failure of the primary database.
12 . The method of claim 11 , further comprising monitoring, by the orchestrator, loading of a computing device of the plurality of computing devices hosting the primary database and instructing the redundancy manager to preemptively failover to the one of the secondary databases in response to the loading indicating a risk of failure.
13 . The method of claim 11 , further comprising monitoring, by the orchestrator, latency of a computing device of the plurality of computing devices hosting the primary database and instructing the redundancy manager to preemptively failover to the one of the secondary databases in response to the latency indicating a risk of failure.
14 . The method of claim 11 , further comprising monitoring, by the orchestrator, temperature of one or more hardware components of a computing device of the plurality of computing devices hosting the primary database and instructing the redundancy manager to preemptively failover to the one of the secondary databases in response to the temperature indicating a risk of failure.
15 . The method of claim 14 , wherein the one or more hardware components include one or more processing devices of the computing device of the plurality of computing devices hosting the primary database.
16 . The method of claim 14 , wherein the one or more hardware components include one or more storage devices of the computing device of the plurality of computing devices hosting the primary database.
17 . The method of claim 11 , further comprising monitoring, by the orchestrator, trends in measured values for a computing device of the plurality of computing devices hosting the primary database.
18 . The method of claim 11 , further comprising selecting the one of the secondary databases as having a least risk of failure of the secondary databases.
19 . The method of claim 11 , wherein the redundancy manager is an instance of PATRONI.
20 . The method of claim 19 , wherein the plurality of databases are POSTGRESQL databases.