Index splitting in distributed databases
In a distributed database, many nodes can store copies, or instances, of the same record. If the record is split on one node, it should be split on the other nodes to maintain consistency, concurrency, and correctness of the data in the distributed database. In some distributed databases, the records are locked during the update process to ensure data integrity. Unfortunately, locking the records can increase latency, especially for larger databases. But if the records aren't locked and a node fails as a record is being split and updated simultaneously, the split and update may not propagate throughout the distributed database, leading to a loss of data integrity. Exchanging messages about the status of record splitting and forwarding updates internally reduces the likelihood of a loss of data integrity due to a node failure.
1. A method of splitting an index atom in a plurality of atoms in a distributed database, the distributed database comprising a plurality of nodes, each node in the plurality of nodes comprising a corresponding processor and a corresponding memory, one node in the plurality of nodes being designated as a chairman for the index atom and including a chairman's instance of the index atom, the method comprising:
splitting, by the chairman, the chairman's instance of the index atom, the chairman's instance of the index atom representing data and/or metadata stored in the distributed database;
transmitting, by the chairman to at least a subset of nodes in the plurality of nodes, instructions to split the index atom, each node in the at least a subset of nodes including a corresponding instance of the index atom;
splitting, by each node in the at least a subset of nodes, the corresponding instance of the index atom into a corresponding instance of a source atom and a corresponding instance of a target atom;
re-transmitting, by each node in the at least a subset of nodes to each other node in the at least a subset of nodes, the instructions to split the index atom;
transmitting, from each node in the at least a subset of nodes to the chairman and to each other node in the at least a subset of nodes, an acknowledgement indicating that the corresponding instance of the index atom has been split;
transmitting, from each node in the at least a subset of nodes to the chairman and to each other node in the at least a subset of nodes, a message indicating that the corresponding instance of the index atom in each node in the at least a subset of nodes has been split;
transmitting, from the chairman to the at least a subset of nodes, a message indicating the index atom has been split;
transmitting, from each node in the at least a subset of nodes to the chairman, an acknowledgement of the message indicating that the index atom has been split;
transmitting, from the chairman to the subset of nodes, a message indicating that the split is complete; and
forwarding, at a first node in the at least a subset of nodes, a message from a first instance of the source atom to a first instance of the target atom,
wherein the first node transmits the acknowledgment to the chairman and to each other node in the plurality of nodes after forwarding the message from the first instance of the source atom to the first instance of the target atom.
2. The method of claim 1 , wherein splitting the chairman's instance of the index atom at the chairman occurs in response to a request to insert a new value into the chairman's instance of the index atom.
3. The method of claim 1 , further comprising:
receiving, by each node in the at least a subset of nodes from each other node in the at least a subset of nodes, the instructions to split the index atom.
4. The method of claim 3 , wherein the first node transmits the acknowledgement to the chairman in response to receiving the instructions to split the index atom from each other node in the at least a subset of nodes.
5. The method of claim 1 , further comprising:
transmitting, by the chairman to another node in the plurality of nodes, a message indicating the index atom has been split, the other node comprising a root atom containing a reference to the index atom.
6. A distributed database comprising:
a plurality of nodes, each node in the plurality of nodes comprising a corresponding processor and a corresponding memory, the plurality of nodes comprising:
a first node containing a first instance of an index atom, the index atom representing data and/or metadata stored in the distributed database;
a second node containing a second instance of the index atom; and
a third node containing a third instance of the index atom,
wherein the first node is configured to split the first instance of the index atom into a first instance of a source atom and a first instance of a target atom and to transmit instructions to split the index atom to the second node and the third node,
the second node is configured to re-transmit the instructions to split the index atom to the third node, to split the second instance of the index atom into a second instance of the source atom and a second instance of the target atom in response to the instructions to split the index atom, to receive a message destined for the second instance of the target atom on the second instance of the source atom and to forward the message from the second instance of the source atom to the second instance of the target atom, and to transmit, to the first node and the third node, an acknowledgement indicating that the second instance of the index atom has been split after the message has been forwarded to the second instance of the target atom, and
the third node is configured to re-transmit the instructions to split the index atom to the second node, to split the third instance of the index atom into a third instance of the source atom and a third instance of the target atom in response to the instructions to split the index atom, and to transmit, to the first node and the second node, an acknowledgement indicating that the third instance of the index atom has been split.
7. The distributed database of claim 6 , wherein:
the first node is configured to send a message to the second node indicating that the index atom has been split in response to receiving the acknowledgement indicating that the second instance of the index atom has been split and to receiving an acknowledgement from the third node indicating that the third instance of the index atom has been split, and
the second node and the third node are configured to send responses to the first node acknowledging the message from the first node.
8. A method of splitting an index atom in a plurality of atoms in a distributed database, the distributed database comprising a plurality of nodes, the index atom representing data and/or metadata stored in the distributed database, each node in the plurality of nodes comprising a corresponding processor and a corresponding memory, one node in the plurality of nodes being designated as a chairman for the index atom and including a chairman's instance of the index atom, the method comprising:
splitting, by the chairman, the chairman's instance of the index atom into a chairman's instance of a source atom and a chairman's instance of a target atom, the chairman's instance of the source atom including values less than a split key value and the chairman's instance of the target atom including values greater than the split key value;
transmitting, by the chairman to at least a subset of nodes in the plurality of nodes, instructions to split the index atom at the split key value, each node in the at least a subset of nodes including a corresponding instance of the index atom;
re-transmitting, by a first node in the at least a subset of nodes to each other node in the at least a subset of nodes, the instructions to split the index atom;
splitting, by the first node in the at least a subset of nodes, a first instance of the index atom into a first instance of a source atom and a first instance of a target atom, the first instance of the source atom including values less than the split key value and the first instance of the target atom including values greater than the split key value;
receiving, by the first node on the first instance of the source atom, a first message referring to a key value greater than the split key value; and
forwarding, by the first node, the first message from the first instance of the source atom to the first instance of the target atom;
transmitting, from the first node to the chairman, an acknowledgement indicating that the first instance of the index atom has been split at the first node;
inserting a reference to the target atom in a root atom referring to the index atom;
transmitting, from the chairman to the at least a subset of nodes, a message indicating the index atom has been split at each node in the at least a subset of nodes; and
receiving, by the first node on the first instance of the target atom, a second message referring to the key value greater than the split key value.
9. The method of claim 8 , further comprising:
tracking, by each node in the at least a subset of nodes, progress of other nodes in the at least a subset of nodes with splitting the index atom.
10. The method of claim 8 , further comprising:
acknowledging, by each node in the at least a subset of nodes, the message from the chairman indicating the index atom has been split at each node in the at least a subset of nodes.