Multi-region, multi-master replication of database tables
A geographically distributed system using a hybrid, multi-master design is disclosed. The system may use global tables, for which both reads and writes may be performed locally, but with a single region serving as a master region. All strongly consistent reads and writes may be sent to the master region, whereas eventually consistent reads and writes may be performed in any region with a replica. The master region may not be statically assigned but may move from region to region. The master region may be assigned manually by client preference or may be assigned and updated automatically based on the time of day or workload characteristics.
1. A system, comprising:
a plurality of computing nodes implementing a plurality of geographically distributed database groups of a distributed database, each of the geographically distributed database groups comprising a plurality of storage nodes including a master node and one or more secondary nodes, wherein each of the geographically distributed database groups hosts a respective replica of the distributed database, wherein each of the geographically distributed database groups performs one or more write requests responsive to receiving the one or more write requests from clients of a cloud computing service providing the distributed database, wherein the plurality of geographically distributed database groups comprises a master group and one or more secondary groups, and wherein the plurality of computing nodes:
receive a write request at a secondary group of the plurality of geographically distributed database groups, wherein the write request comprises an update to at least a portion of the distributed database;
determine, by the secondary group, whether to perform or forward the received write request according to a first type of write consistency associated with the write request, the first type of write consistency being one of a plurality of different types of write consistency provided by the distributed database; and
responsive to a determination by the secondary group that the first type of write consistency is strong consistency that indicates that the write request should be forwarded instead of performed, forward the write request from the secondary group to the master group of the plurality of geographically distributed database groups.
2. The system of claim 1 , wherein the plurality of computing nodes further:
receive a second write request at the secondary group of the plurality of geographically distributed database groups;
perform the second write request at the secondary group of the plurality of geographically distributed database groups responsive to the second write request being associated with a second type of write consistency among a plurality of types of write consistency provided by the distributed database, wherein the second type of write consistency is eventual consistency.
3. The system of claim 2 , wherein the plurality of computing nodes further:
synchronize, subsequent to the performance of the second write request, the respective replicas of the master and secondary groups of the plurality of geographically distributed database groups, wherein the synchronization is configured to converge the respective replicas such that the result of the second write request is transferred to the master group of the plurality of geographically distributed database groups.
4. The system of claim 1 , wherein the plurality of computing nodes further:
receive a request to promote a secondary group of the plurality of geographically distributed database groups to be the master group, wherein the master group is identified by a master lease;
create, responsive to the request to promote the secondary group, a transition lease identifying the secondary group of the plurality of geographically distributed database groups as the next master group.
5. A method, comprising:
receiving a write request at a first node of a plurality of nodes each hosting respective replicas of a distributed database and each performing write requests responsive to receiving the write requests from clients of the distributed database, wherein the write request comprises an update to at least a portion of the distributed database;
determining, by the first node, whether to perform or forward the received write request according to a first type of write consistency associated with the write request, the first type of write consistency being one of a plurality of different types of write consistency provided by the distributed database; and
forwarding the write request to a second node of the plurality of nodes, the second node hosting a master replica of the distributed database;
wherein said forwarding is performed responsive to a determination that the first type of write consistency indicates that the write request should be forwarded instead of performed and responsive to the first node not hosting the master replica.
6. The method of claim 5 , further comprising:
synchronizing respective replicas of two or more of the plurality of nodes including the second node hosting the master replica of the distributed database, wherein the synchronization converges the respective replicas such that the data is consistent between the respective replicas, and wherein a conflict resolution policy used in the synchronization is associated with a client application of the distributed database.
7. The method of claim 5 , further comprising:
receiving a second write request at the first node of the plurality of nodes;
performing the second write request at the first node of the plurality of nodes responsive to the second write request being associated with a second type of write consistency among a plurality of types of write consistency provided by the distributed database.
8. The method of claim 7 , further comprising:
synchronizing, subsequent to the performance of the second write request, the respective replicas of the first and second nodes of the plurality of nodes, wherein the synchronization converges the respective replicas such that the result of the second write request is transferred to the master replica of the second node of the plurality of nodes.
9. The method of claim 5 , further comprising:
receiving a request to promote a replica hosted at a third node of the plurality of nodes to be the master replica, wherein the master replica is identified by a master lease;
creating, responsive to the request to promote the replica hosted at a third node, a transition lease identifying the replica hosted at the third node as the next master replica.
10. The method of claim 5 , further comprising converting the transition lease, responsive to the expiring of the master lease, into a new master lease.
11. The method of claim 7 , wherein performing the second write request at the first node of the plurality of storage groups comprises:
scheduling the second write request to be performed at the first node of the plurality of nodes; and
performing the second write request at the first node of the plurality of nodes.
12. The method of claim 5 , wherein the distributed database is a geographically distributed database, wherein the first and second nodes of the plurality of nodes are implemented as part of regional storage groups comprising one or more nodes, and wherein the first and second nodes reside in different geographic regions.
13. A non-transitory computer-readable medium storing program instructions, that when executed by one or more processors, cause the one or more processors to:
receive a write request at a first node of a plurality of nodes each hosting respective replicas of a distributed database and each performing write requests responsive to receiving the write requests from clients of the distributed database, wherein the write request comprises an update to at least a portion of the distributed database;
determine, by the first node, whether to perform or forward the received write request according to a first type of write consistency associated with the write request, the first type of write consistency being one of a plurality of different types of write consistency provided by the distributed database; and
forward the write request to a second node of the plurality of nodes, the second node hosting a master replica of the distributed database;
wherein said forwarding is performed responsive to a determination that the first type of write consistency indicates that the write request should be forwarded instead of performed and responsive to the first node not hosting the master replica.
14. The non-transitory computer-readable medium of claim 13 , wherein the program instructions, when executed by the one or more processors, cause the one or more processors to further:
synchronize respective replicas of two or more of the plurality of nodes including the second node hosting a master replica of the distributed database, wherein the synchronization converges the respective replicas such that the data is consistent between the respective replicas, and wherein a conflict resolution policy used in the synchronization is associated with a client application of the distributed database.
15. The non-transitory computer-readable medium of claim 13 , wherein the program instructions, when executed by the one or more processors, cause the one or more processors to further:
receive a second write request at the first node of the plurality of nodes;
perform the second write request at the first node of the plurality of nodes responsive to the second write request being associated with a second type of write consistency among a plurality of types of write consistency provided by the distributed database.
16. The non-transitory computer-readable medium of claim 15 , wherein the program instructions, when executed by the one or more processors, cause the one or more processors to further:
synchronize, subsequent to the performance of the second write request, the respective replicas of the first and second nodes of the plurality of nodes, wherein the synchronization converges the respective replicas such that the result of the second write request is transferred to the master replica of the second node of the plurality of nodes.
17. The non-transitory computer-readable medium of claim 15 , wherein the program instructions, when executed by one or more processors, cause the one or more processors to further schedule the second write request to be performed at a storage node associated with the first node of the plurality of nodes to perform the second write request at the first node of the plurality of nodes.
18. The non-transitory computer-readable medium of claim 13 , wherein the program instructions, when executed by the one or more processors, cause the one or more processors to further:
receive a request to promote a replica hosted at a third node of the plurality of computing nodes to be the master replica, wherein the master replica is identified by a master lease;
create, responsive to the request to promote the replica hosted at a third node, a transition lease identifying the replica hosted at the third node as the next master replica.
19. The non-transitory computer-readable medium of claim 18 , wherein the program instructions when executed by the one or more processors cause the one or more processors to further convert the transition lease, responsive to the expiring of the master lease, into a new master lease.
20. The non-transitory computer-readable medium of claim 13 , wherein the distributed database is a geographically distributed database, wherein the first and second nodes of the plurality of nodes are regional storage groups comprising one or more nodes, and wherein the first and second nodes reside in different geographic regions.