Maintaining partition-tolerant distributed metadata

1. A computer-implemented method, comprising:

receiving, from a new node, a request indicating the new node is joining a cluster of nodes;

transmitting, to the new node in response to the request, an identifier of a delegate node within the cluster;

receiving, from the delegate node, a request to maintain a synchronization point of the delegate node while the delegate node creates a snapshot of cluster metadata and transmits the snapshot to the new node;

receiving, from the new node, an indication that the new node has received the snapshot and is ready to join the cluster;

determining that the cluster has synchronized metadata distributed within the cluster

following the synchronization point of the delegate node and prior to receiving the indication that the new node is ready to join the cluster; and

in response to the determination, transmitting an acknowledgment of the indication that the new node is ready to join the cluster and to cause the new node to transition to an active state within the cluster.

2. The computer-implemented method of claim 1 , wherein the request indicating the new node is joining the cluster of nodes is a request for a snapshot of metadata synchronized within the cluster.

3. The computer-implemented method of claim 1 , wherein the metadata is a plurality of key/value pairs of metadata for storage within the cluster.

4. The computer-implemented method of claim 1 , further comprising:

selecting the delegate node by applying a round robin selection algorithm to active nodes within the cluster.

5. The computer-implemented method of claim 1 , further comprising:

receiving, from the new node, metadata entries the new node determined were not included in the snapshot and stored by the new node prior to the new node transmitting the request indicating the new node is joining the cluster; and

broadcasting, to the cluster, the metadata entries received from the new node, wherein determining that the cluster has synchronized metadata distributed within the cluster includes determining that the metadata entries received from the new node have been synchronized across the cluster.

6. The computer-implemented method of claim 1 , wherein the new node was previously a part of the cluster and is rejoining the cluster.

7. The computer-implemented method of claim 1 , wherein the synchronization point of the delegate node is a sequence number of a last metadata entry received by the delegate node, the sequence number indicating not to drain a retransmission queue of metadata entries corresponding to subsequent sequence numbers until the new node acknowledges the metadata entries corresponding to subsequent sequence numbers.

8. The computer-implemented method of claim 1 , wherein receiving the indication that the new node is ready to join the cluster includes receiving a key/value pair including a unique identifier for the new node.

9. A non-transitory computer-readable medium storing instructions, which when executed by a processing device, cause the processing device to perform a method comprising:

receiving, from a new node, a request indicating the new node is joining a cluster of nodes;

transmitting, to the new node in response to the request, an identifier of a delegate node within the cluster;

receiving, from the delegate node, a request to maintain a synchronization point of the delegate node while the delegate node creates a snapshot of cluster metadata and transmits the snapshot to the new node;

receiving, from the new node, an indication that the new node has received the snapshot and is ready to join the cluster;

determining that the cluster has synchronized metadata distributed within the cluster

following the synchronization point of the delegate node and prior to receiving the indication that the new node is ready to join the cluster; and

in response to the determination, transmitting an acknowledgment of the indication that the new node is ready to join the cluster and to cause the new node to transition to an active state within the cluster.

10. The non-transitory computer-readable medium of claim 9 , wherein the request indicating the new node is joining the cluster of nodes is a request for a snapshot of metadata synchronized within the cluster.

11. The non-transitory computer-readable medium of claim 9 , wherein the metadata is a plurality of key/value pairs of metadata for storage within the cluster.

12. The non-transitory computer-readable medium of claim 9 , the method further comprising:

selecting the delegate node by applying a round robin selection algorithm to active nodes within the cluster.

13. The non-transitory computer-readable medium of claim 9 , the method further comprising:

receiving, from the new node, metadata entries the new node determined were not included in the snapshot and stored by the new node prior to the new node transmitting the request indicating the new node is joining the cluster; and

broadcasting, to the cluster, the metadata entries received from the new node, wherein determining that the cluster has synchronized metadata distributed within the cluster includes determining that the metadata entries received from the new node have been synchronized across the cluster.

14. The non-transitory computer-readable medium of claim 9 , wherein the new node was previously a part of the cluster and is rejoining the cluster.

15. The non-transitory computer-readable medium of claim 9 , wherein the synchronization point of the delegate node is a sequence number of a last metadata entry received by the delegate node, the sequence number indicating not to drain a retransmission queue of metadata entries corresponding to subsequent sequence numbers until the new node acknowledges the metadata entries corresponding to subsequent sequence numbers.

16. The non-transitory computer-readable medium of claim 9 , wherein receiving the indication that the new node is ready to join the cluster includes receiving a key/value pair including a unique identifier for the new node.

17. A master node comprising:

a processing device; and

a memory coupled to the processing device, the memory storing instructions which, when executed by the processing device, cause the master node to:

receive, from a new node, a request indicating the new node is joining a cluster of nodes;

transmit, to the new node in response to the request, an identifier of a delegate node within the cluster;

receive, from the delegate node, a request to maintain a synchronization point of the delegate node while the delegate node creates a snapshot of cluster metadata and transmits the snapshot to the new node;

receive, from the new node, an indication that the new node has received the snapshot and is ready to join the cluster;

determine that the cluster has synchronized metadata distributed within the cluster following the synchronization point of the delegate node and prior to receiving the indication that the new node is ready to join the cluster; and

in response to the determination, transmit an acknowledgment of the indication that the new node is ready to join the cluster and cause the new node to transition to an active state within the cluster.

18. The master node of claim 17 , wherein the request indicating the new node is joining the cluster of nodes is a request for a snapshot of metadata synchronized within the cluster.

19. The master node of claim 17 , wherein the metadata is a plurality of key/value pairs of metadata for storage within the cluster.

20. The master node of claim 17 , wherein the instructions further cause the master node to:

receive, from the new node, metadata entries the new node determined were not included in the snapshot and stored by the new node prior to the new node transmitting the request indicating the new node is joining the cluster; and

broadcast, to the cluster, the metadata entries received from the new node, wherein determining that the cluster has synchronized metadata distributed within the cluster includes determining that the metadata entries received from the new node have been synchronized across the cluster.