ENSURING HIGH AVAILABLITY OF REPLICATED DATABASE MANAGEMENT SYSTEMS DURING UPGRADES
An online system, such as a multi-tenant system ensures high availability of systems, for example, database management systems. The online system replicates the databases across multiple datacenters including: (1) a master node that receives read and write requests (2) a read-replica that receives only read requests and (3) a spare node that does not receive requests but acts as standby for high availability. One or more application servers may send read and write requests to the databases. The system performs a sweep of upgrades of the database nodes and also performs traffic quiescing of the requests received from the application servers to redirect the traffic across the database nodes as the upgrade sweep is orchestrated. The sweep of upgrades ensures that the availability of the database management system to the end users is maximized during the upgrade process.
1 . A computer implemented method for upgrading database management systems, the method comprising:
receiving a request to upgrade a replicated database management system comprising: a first node configured as a master node, a second node configured as a read-replica node, and a third node configured as a spare node, wherein the master node processes write requests received by the replicated database management system and read requests are distributed across the master node and the read-replica node;
upgrading the third node configured as the spare node;
subsequent to upgrading the third node, configuring the third node as the read-replica node and the second node as the spare node;
subsequent to configuring the second node as the spare node, upgrading the second node;
subsequent to upgrading the second node, configuring the first node as the spare node and one of the second node or the third node as the master node; and
subsequent to configuring the first node as the spare node, upgrading the first node.
2 . The method of claim 1 , wherein configuring the third node as the read-replica node and the second node as the spare node, comprises:
quiescing read requests directed to the second node; and
distributing read requests across the first node and the third node.
3 . The method of claim 1 , wherein configuring the first node to be the spare node and one of the second node or the third node to be the master node, comprises:
quiescing read and write requests directed to the first node; and
subsequent to quiescing the read and write request directed to the first node, directing write requests to the master node and distributing read requests across second and the third node.
4 . The method of claim 1 , wherein the read and write requests are received from a set of application servers, wherein quiescing requests directed to a node comprises:
sending a request to each of the application servers to quiesce requests; and
waiting for each of the set of application servers to send an acknowledgement message indicating a completion of quiescing of requests.
5 . The method of claim 1 , wherein the nodes of the replicated database management system are distributed across three data centers wherein the three data centers are situated in different physical locations.
6 . The method of claim 1 , wherein the spare node is used in case of failure of one of the master node or the read-replica node.
7 . The method of claim 1 , wherein the upgrade of a node comprises one or more of: installing a patch or upgrading to a new version of a software for the database management system.
8 . The method of claim 1 , wherein the replicated database management system is deployed on a cloud platform, the method further comprising:
receiving a cloud platform image for a new version of a software for the database management system, wherein the cloud platform image is used for upgrading any node.
9 . The method of claim 8 , wherein a database management system is stored using:
an instructions storage unit storing instructions of the database management system for processing data of a database; and
a data storage unit storing data of the database, wherein upgrading the database management system comprises installing new version of software for the database management system on a new instructions storage unit and providing the new instructions storage unit with access to the data storage unit.
10 . A non-transitory computer readable storage medium for storing instructions that when executed by a computer processor cause the computer processor to perform steps comprising:
receiving a request to upgrade a replicated database management system comprising: a first node configured as a master node, a second node configured as a read-replica node, and a third node configured as a spare node, wherein the master node processes write requests received by the replicated database management system and read requests are distributed across the master node and the read-replica node;
upgrading the third node configured as the spare node;
subsequent to upgrading the third node, configuring the third node as the read-replica node and the second node as the spare node;
subsequent to configuring the second node as the spare node, upgrading the second node;
subsequent to upgrading the second node, configuring the first node as the spare node and one of the second node or the third node as the master node; and
subsequent to configuring the first node as the spare node, upgrading the first node.
11 . The non-transitory computer readable storage medium of claim 10 , wherein configuring the third node as the read-replica node and the second node as the spare node, comprises:
quiescing read requests directed to the second node; and
distributing read requests across the first node and the third node.
12 . The non-transitory computer readable storage medium of claim 10 , wherein configuring the first node to be the spare node and one of the second node or the third node to be the master node comprises:
quiescing read and write requests directed to the first node; and
subsequent to quiescing the read and write request directed to the first node, directing write requests to the master node and distributing read requests across second and the third node.
13 . The non-transitory computer readable storage medium of claim 10 , wherein the read and write requests are received from a set of application servers, wherein quiescing requests directed to a node comprises:
sending a request to each of the application servers to quiesce requests; and
waiting for each of the set of application servers to send an acknowledgement message indicating a completion of quiescing of requests.
14 . The non-transitory computer readable storage medium of claim 10 , wherein the nodes of the replicated database management system are distributed across three data centers wherein the three data centers are situated in different physical locations.
15 . The non-transitory computer readable storage medium of claim 10 , wherein the replicated database management system is deployed on a cloud platform, the instructions further causing the computer processor to perform steps comprising:
receiving a cloud platform image for a new version of a software for the database management system, wherein the cloud platform image is used for upgrading any node.
16 . The non-transitory computer readable storage medium of claim 15 , wherein a database management system is stored using:
an instructions storage unit storing instructions of the database management system for processing data of a database; and
a data storage unit storing data of the database, wherein upgrading the database management system comprises installing new version of software for the database management system on a new instructions storage unit and providing the new instructions storage unit with access to the data storage unit.
17 . A computer system comprising:
a computer processor; and
a non-transitory computer readable storage medium for storing instructions that when executed by the computer processor cause the computer processor to perform steps comprising:
receiving a request to upgrade a replicated database management system comprising: a first node configured as a master node, a second node configured as a read-replica node, and a third node configured as a spare node, wherein the master node processes write requests received by the replicated database management system and read requests are distributed across the master node and the read-replica node;
upgrading the third node configured as the spare node;
subsequent to upgrading the third node, configuring the third node as the read-replica node and the second node as the spare node;
subsequent to configuring the second node as the spare node, upgrading the second node;
subsequent to upgrading the second node, configuring the first node as the spare node and one of the second node or the third node as the master node; and
subsequent to configuring the first node as the spare node, upgrading the first node.
18 . The computer system of claim 17 , wherein configuring the third node as the read-replica node and the second node as the spare node, comprises:
quiescing read requests directed to the second node; and
distributing read requests across the first node and the third node.
19 . The computer system of claim 17 , wherein configuring the first node to be the spare node and one of the second node or the third node to be the master node comprises:
quiescing read and write requests directed to the first node;
directing write requests to the master node; and
distributing read requests across second and the third node.
20 . The computer system of claim 17 , wherein the read and write requests are received from a set of application servers, wherein quiescing requests directed to a node comprises:
sending a request to each of the application servers to quiesce requests; and
waiting for each of the set of application servers to send an acknowledgement message indicating a completion of quiescing of requests.