Atomic clustering operations for managing a partitioned cluster online
A set of six atomic primitives are provided for a database management system that may be used in combination to provide all of the common features and functions of a clustered database, including data failover, auto-healing, and elastic scaling. These six atomic primitives include CREATE, DROP, DETACH, ATTACH, COPY, and PROMOTE. Of particular note, it is shown that by maintaining appropriate metadata, including the status of each instance of each partition in the cluster, the versatility and reliability of this set of primitives is sufficient to implement each of the aforementioned data failover, auto-healing, and elastic scaling functions with high efficiency using a minimal number of these primitives. Each primitive is atomic (such that the cluster clearly in one state or another) and online (a workload of reads and writes is uninterrupted while the primitive runs), and each primitive is scoped to a single partition of data, thereby enabling parallel processing.
1. A database management system that comprises:
a cluster of data nodes, each data node being able to store data of a plurality of partitions of one or more databases;
an aggregator that manages the cluster and maintains metadata of the cluster that maintains a list of instances of database partitions on nodes, each instance being able to store the data of the corresponding database partition;
wherein the aggregator is configured to include a set of primitives that are atomic, online, and partition specific, the set of primitives including:
CREATE <Partition> ON <Node>, which creates a new primary instance of the Partition on the Node;
DROP <Partition> ON <Node>, which removes the instance of the Partition from the Node;
DETACH <Partition> ON <Node>, which identifies the instance of the Partition on the Node as being detached from the Node, without deleting the data in the instance;
ATTACH <Partition> ON <Node>, which identifies the instance of the Partition on the Node as re-instantiated on the Node, without changing the data in the instance;
COPY <Partition> ON <Node>, which creates a new follower instance of the Partition on the Node; and
PROMOTE <Partition> ON <Node>, which identifies the instance of the Partition on the Node as a primary partition; and
wherein the database management system is configured to provide data failover, auto-recovery, and elastic scaling using only the set of primitives.
2. The system of claim 1 , wherein the DETACH, ATTACH, and PROMOTE primitives process and affect only the metadata of the cluster.
3. The system of claim 1 , wherein the CREATE, DROP, and COPY primitives affect only the metadata of the cluster and metadata of the nodes.
4. The system of claim 1 , wherein the metadata of the cluster includes a status associated with each instance of each partition in the cluster, the status including whether the instance is detached from the cluster.
5. The system of claim 4 , wherein the status includes whether data in the instance is valid.
6. The system of claim 1 , wherein the new primary instance and new follower instance are instances that contain no data when created by the CREATE and COPY primitives.
7. The system of claim 1 , wherein the system provides the data failover using the DETACH, PROMOTE, and CREATE primitives, and the auto-recovery using the ATTACH primitive.
8. The system of claim 1 , wherein the system provides the elastic scaling via a rebalance function that uses the COPY, PROMOTE, and DROP primitives.
9. A non-transitory computer readable medium that includes a program that, when executed by a processor causes the processor to:
manage a cluster of data nodes, each data node being able to store data of a plurality of partitions of a database; and
maintain metadata of the cluster that includes a list of instances of database partitions on nodes, each instance being able to store the data of the corresponding database partition;
wherein the program enables the processor to provide data failover, auto-recovery, and elastic scaling using only a set of primitives that include:
CREATE <Partition> ON <Node>, which creates a new primary instance of the Partition on the Node;
DROP <Partition> ON <Node>, which removes the instance of the Partition from the Node;
DETACH <Partition> ON <Node>, which identifies the instance of the Partition on the Node as being detached from the Node, without deleting the data in the instance;
ATTACH <Partition> ON <Node>, which identifies the instance of the Partition on the Node as re-instantiated on the Node, without changing the data in the instance;
COPY <Partition> ON <Node>, which creates a new follower instance of the Partition on the Node; and
PROMOTE <Partition> ON <Node>, which identifies the instance of the Partition on the Node as a primary partition; and
wherein each of these primitives are performed by modifying the metadata associated with the Partition, independent of the data stored at the Node.
10. The medium of claim 9 , wherein each primitive of the set of primitives is atomic and online.
11. The medium of claim 9 , wherein the DETACH, ATTACH, and PROMOTE primitives process and affect only the metadata of the cluster.
12. The medium of claim 9 , wherein the CREATE, DROP, and COPY primitives affect only the metadata of the cluster and metadata of the nodes.
13. The medium of claim 9 , wherein the metadata of the cluster includes a status associated with each instance of each partition in the cluster, the status including whether the instance is detached from the cluster.
14. The medium of claim 13 , wherein the status includes whether data in the instance is valid.
15. The medium of claim 9 , wherein the new primary instance and new follower instance are instances that contain no data when created by the CREATE and COPY primitives.
16. The medium of claim 9 , wherein the program causes the processing system to provide the data failover using only the DETACH, PROMOTE, and CREATE primitives, and the auto-recovery using only the ATTACH primitive.
17. The medium of claim 9 , wherein the program causes the processing system to provide the elastic scaling via a rebalance function that uses only the COPY, PROMOTE, and DROP primitives.