Minimizing false negative and duplicate health monitoring alerts in a dual master shared nothing database appliance
View Patent ↗A primary master node and a standby master node monitor the health of a shared nothing database appliance to afford high availability while minimizing false negatives and duplicate alerts by executing continuously in parallel complimentary processes that determine whether the database is running, and which master node is the active database master node. The active database master node monitors the health of the components of the database appliance by polling each component to detect failures and warnings, and the other master node monitors the status of the active master node. Upon detecting a failure of the active master node, the other node takes over health monitoring. If the database is not running, the designated primary master node performs health monitoring.
1. A method of monitoring the health of a database appliance comprising a database distributed on a plurality of database nodes, and having redundant master nodes including a primary master node and a standby master node, the database being active on and controlled by only one of said redundant master nodes at a time, the method comprising:
executing concurrently in parallel and independently on both said redundant master nodes a database monitoring process, said database monitoring process comprising a resolution process and a health monitor process, the resolution process resolving on which one of said redundant master nodes said database is active at said time, said one node being designated the primary master node, and confirming that said primary master node is executing said health monitor process to monitor hardware and software components of said database and report alerts, the other redundant master node being the standby master node and not issuing alerts;
resolving by executing said resolution process in parallel by said primary master node and said standby master node whether the database is running on said primary master node, including:
attempting, by the primary master node, a first login to the database on the primary master node; and
attempting concurrently, by the standby master node, a second login to the database on the primary master node;
upon said first and second logins being successful, resolving by the resolution process on the primary master node that the database is running on the primary master node and that the primary master node is executing said health monitor process of hardware and software components of the database;
monitoring by the standby master node the status of the primary master node to detect a failure of the primary master node, including:
attempting, by the standby master node, a third login to the database on the primary master node after a first predetermined period of time;
upon identifying that the third login attempt is unsuccessful, determining that the primary master node has failed based on the unsuccessful third login attempt; and
upon determining said failure of the primary master node by the standby master node:
attempting, by the standby master node, a fourth login to the database on the standby master node;
upon the fourth login attempt by the standby master node being successful, determining that the database is active on said standby master node; and
executing said health monitor process of said components of said database by the standby master node in response to determining that the fourth login attempt was successful.
2. The method of claim 1 , where attempting said first login comprises issuing a database query to the primary master node, and determining that the database is running upon receiving a response to the query.
3. The method of claim 1 further comprising waiting a second predetermined period of time following said executing said health monitor process by said primary master node, and repeating said determining and health monitor process by said primary master node.
4. The method of claim 1 , wherein said first predetermined period of time is of the order of about 1-5 minutes and said second predetermined period of time is of the order of about 1-5 seconds.
5. The method of claim 1 , further comprising:
attempting, by the standby master node, a fifth login to the database on the standby master node;
determining that said fifth login to the database on the standby master node is unsuccessful;
in response to determining that said fifth login attempt was unsuccessful, attempting, by the standby master node, to contact a health monitor process on the primary master node; and, if said contact is unsuccessful, performing said health monitor process by said standby master node.
6. The method of claim 1 , wherein said health monitor process comprises polling said hardware and software components to determine their status, and issuing alerts upon detecting a failure or a warning.
7. The method of claim 1 further comprising determining, upon said first and second logins being unsuccessful, that said database is not operating, and issuing an alert by one of said redundant master nodes.
8. Non-transitory computer readable media storing executable instructions for controlling the operation of a computer to perform health monitoring of a database appliance that includes a database distributed on a plurality of database nodes, and having redundant master nodes including a primary master node and a standby master node, the database being active on and controlled by only one of said redundant master nodes at a time, comprising instructions for:
executing concurrently in parallel and independently on both said redundant master nodes a database monitoring process, said database monitoring process comprising a resolution process and a health monitor process, the resolution process resolving on which one of said redundant master nodes said database is active at said time, said one node being designated the primary master node, and confirming that said primary master node is executing said health monitor process to monitor hardware and software components of said database and report alerts, the other redundant master node being the standby master node and not issuing alerts;
resolving by executing said resolution process in parallel by said primary master node and said standby master node whether the database is running on said primary master node, including:
attempting, by the primary master node, a first login to the database on the primary master node; and
attempting concurrently, by the standby master node, a second login to the database on the primary master node;
upon said first and second logins being successful, resolving by the resolution process on the primary master node that the database is running on the primary master node and that the primary master node is executing said health monitor process of hardware and software components of the database;
monitoring by the standby master node the status of the primary master node to detect a failure of the primary master node, including:
attempting, by the standby master node, a third login to the database on the primary master node after a first predetermined period of time;
upon identifying that the third login attempt is unsuccessful,
determining that the primary master node has failed based on the unsuccessful third login attempt; and
upon determining said failure of the primary master node by the standby master node:
attempting, by the standby master node, a fourth login to the database on the standby master node;
upon the fourth login attempt by the standby master node being successful, determining that the database is active on said standby master node; and
executing said health monitor process of said components of said database by the standby master node in response to determining that the fourth login attempt was successful.
9. Non-transitory computer readable media according to claim 8 , wherein said instructions for attempting said first login comprise instructions for issuing a database query to the primary master node, and determining that the database is running upon receiving a response to the query.
10. Non-transitory computer readable media according to claim 8 further comprising instructions for waiting a second predetermined period of time following said executing said health monitor process by said primary master node, and repeating said determining and health monitor process by said primary master node.
11. Non-transitory computer readable media according to claim 8 , further comprising instructions for:
attempting, by the standby master node, a fifth login to the database on the standby master node;
determining that said fifth login to the database on the standby master node is unsuccessful;
in response to determining that said fifth login attempt was unsuccessful, attempting, by the standby master node, to contact a health monitor process on the primary master node; and, if said contact is unsuccessful, performing said health monitor process by said standby master node.
12. Non-transitory computer readable media according to claim 8 , wherein said health monitor process comprises instructions for polling said hardware and software components to determine their status, and for issuing alerts upon detecting a failure or a warning.
13. A method of monitoring the health of a database appliance comprising a database distributed on a plurality of database nodes, and having redundant master nodes including a primary master node and a standby master node, the database being active on and controlled by only one of said redundant master nodes at a time, the method comprising:
executing concurrently in parallel and independently on both said redundant master nodes a database monitoring process, said database monitoring process comprising a resolution process and a health monitor process, the resolution process resolving on which one of said redundant master nodes said database is active at said time, said one node being designated the primary master node, and confirming that said primary master node is executing said health monitor process to monitor hardware and software components of said database and report alerts, the other redundant master node being the standby master node and not issuing alerts;
resolving by executing said resolution process in parallel and concurrently by said redundant master nodes on which one of said redundant master nodes the database is currently active;
designating said one redundant master node on which the database is currently active as the primary master node, and designating the other redundant master node as the standby master node;
executing substantially continuously by said primary master node and by said standby master node said resolution processes to confirm that the database remains active on the primary master node;
upon confirming the database remains active on the primary master node, executing by the primary master node the health monitor process to monitor the health of said database, and executing concurrently by said standby master node said health monitor process to monitor the health of said primary master node;
upon the health monitor process executing on the primary master node determining that the database has failed, issuing a first alert and executing by said standby master node said health monitor process of said database to confirm said failure;
upon determining by said primary master node said database failure, issuing a second alert; and
otherwise upon the health monitor process on the standby master node determining that either or both the primary master node or the database has failed, issuing by the standby master node a third alert.
14. The method of claim 13 , wherein said resolving comprises issuing by each of said redundant master nodes a database query to both of said redundant master nodes, and resolving the master node on which the database is active by a response to said query.
15. The method of claim 13 , wherein said standby master node confirms the failure of the database on the primary master node by attempting unsuccessfully to login to the database on the primary master node.
16. The method of claim 13 , wherein said standby master node monitors the health of said primary master node by attempting to contact the health monitor process on the primary master node, and determines that the primary master node has failed if said contact is unsuccessful.