Commissioning and decommissioning metadata nodes in a running distributed data storage system
In a running distributed data storage system that actively processes I/Os, metadata nodes are commissioned and decommissioned without taking down the storage system and without introducing interruptions to metadata or payload data I/O. The inflow of reads and writes continues without interruption even while new metadata nodes are in the process of being added and/or removed and the strong consistency of the system is guaranteed. Commissioning and decommissioning nodes within the running system enables streamlined replacement of permanently failed nodes and advantageously enables the system to adapt elastically to workload changes. An illustrative distributed barrier logic (the “view change barrier”) controls a multi-state process that controls a coordinated step-wise progression of the metadata nodes from an old view to a new normal. Rules for I/O handling govern each state until the state machine loop has been traversed and the system reaches its new normal.
1. A system comprising:
a plurality of storage service nodes, wherein each key in a set of keys is owned by exactly one storage service node among the plurality of storage service nodes;
wherein among the plurality of storage service nodes, a first storage service node is configured to: own a first range of keys within the set of keys, receive read requests and write requests for metadata that is associated with the first range of keys, and store and maintain first metadata files at the first storage service node, wherein each first metadata file is associated with at least one key within the first range of keys;
wherein among the plurality of storage service nodes, at least one storage service node is configured to: control commissioning of a second storage service node into the system, wherein the commissioning is performed without interrupting servicing, by the system, of read requests from and write requests to any of the plurality of storage service nodes;
wherein the system further comprises the second storage service node, which is configured to: claim a second range of keys from the set of keys, wherein the second range of keys comprises at least part of the first range of keys; and
wherein after the commissioning, the second storage service node is further configured to: receive read requests and write requests for metadata associated with the second range of keys, and store and maintain, at the second storage service node, any of the first metadata files that are associated with at least one key within the second range of keys.
2. The system of claim 1 , wherein the commissioning of the second storage service node is initiated based on one or more of: detecting that the second storage service node has been activated, and detecting that the second storage service node claimed the second range of keys.
3. The system of claim 1 , wherein before claiming the second range of keys from the set of keys, the second storage service node is activated without having metadata files associated with any key in the set of keys.
4. The system of claim 1 , wherein during the commissioning, the second range of keys becomes owned by the second storage service node, and further wherein after the commissioning, the first storage service node no longer receives read requests and write requests for metadata associated with the second range of keys, and no longer maintains, at the first storage service node, the first metadata files that are associated with the at least one key within the second range of keys.
5. The system of claim 1 , wherein the commissioning comprises a progression of operations that are configured to guarantee strong consistency within the system.
6. The system of claim 1 , wherein the commissioning is based on a state machine that controls a progression of operations that are configured to guarantee strong consistency within the system.
7. The system of claim 1 , wherein the set of keys are unique identifiers that are used for ensuring strong consistency of data and metadata within the system.
8. The system of claim 1 , wherein the commissioning comprises re-distributing the set of keys among the second storage service node and one or more storage service nodes of the plurality of storage service nodes, including re-distributing the first range of keys between the first storage service node and the second storage service node.
9. The system of claim 1 , wherein the first storage service node is configured to execute a first instance of a metadata subsystem, and wherein the second storage service node is configured to execute a second instance of the metadata subsystem, and wherein the commissioning comprises re-distributing the set of keys among storage service nodes that execute a respective instance of the metadata subsystem, including re-distributing the first range of keys between the first instance of the metadata subsystem and the second instance of the metadata subsystem.
10. The system of claim 1 , wherein before the second storage service node is activated, the first storage service node operates according to a current view that indicates that the first range of keys is associated with the first storage service node, and wherein after the second storage service node is activated, the first storage service node acquires a new view that indicates that the second range of keys is associated with the second storage service node.
11. The system of claim 10 , wherein the at least one storage service node is configured to: inform the first storage service node and the second storage service node that the first storage service node and the second storage service node have successfully completed copying any of the first metadata files that are associated with at least one key within the second range of keys from the first storage service node to the second storage service node, discard the current view, and establish the new view as the current view for the system, wherein establishment of the new view signifies completion of the commissioning of the second storage service node.
12. The system of claim 1 , wherein the at least one storage service node is further configured to control decommissioning of the second storage service node from the system without interruption of read requests from and write requests to any of the plurality of storage service nodes,
wherein the decommissioning re-distributes ownership of keys within the second range of keys to at least one other storage service node within the system that is distinct from the second storage service node; and
wherein based on completion of the decommissioning, the second storage service node is not authorized to process any read requests and any write requests within the system.
13. A system comprising:
a plurality of storage service nodes, wherein each key in a set of keys is owned by exactly one storage service node among the plurality of storage service nodes;
wherein among the plurality of storage service nodes, a first storage service node is configured to: own a first range of keys within the set of keys, service read requests and write requests for metadata that is associated with the first range of keys, and store and maintain first metadata files at the first storage service node, wherein each first metadata file is associated with one or more key within the first range of keys;
wherein among the plurality of storage service nodes, at least one storage service node is configured to: control commissioning of a second storage service node into the system, wherein the commissioning is based on a state machine that controls a progression of operations, wherein the progression of operations enables the system to continue servicing read requests from and write requests to any of the plurality of storage service nodes, and wherein the progression of operations guarantees strong consistency within the system; and
wherein the system further comprises the second storage service node, which is configured to: claim a second range of keys from the set of keys, wherein the second range of keys comprises at least part of the first range of keys;
wherein the commissioning comprises copying, from the first storage service node to the second storage service node, first metadata files that are associated with one or more keys within the second range of keys; and
wherein after the commissioning, the second storage service node is further configured to: service read requests and write requests for metadata associated with the second range of keys, and store and maintain, at the second storage service node, the first metadata files that are associated with the one or more keys within the second range of keys.
14. The system of claim 13 , wherein the commissioning of the second storage service node is initiated based on one or more of: detecting that the second storage service node has been activated, and detecting that the second storage service node claimed the second range of keys.
15. The system of claim 13 , wherein before claiming the the second range of keys from the set of keys, the second storage service node is activated without having metadata files associated with any key in the set of keys, and wherein during the commissioning, the second range of keys becomes owned by the second storage service node.
16. The system of claim 13 , wherein after the commissioning, the first storage service node no longer services read requests and write requests for metadata associated with the second range of keys, and no longer maintains, at the first storage service node, the first metadata files that are associated with the one or more keys within the second range of keys.
17. The system of claim 13 , wherein the commissioning comprises re-distributing the set of keys among the second storage service node and one or more storage service nodes of the plurality of storage service nodes, including re-distributing the first range of keys between the first storage service node and the second storage service node.
18. The system of claim 13 , wherein the first storage service node is configured to execute a first instance of a metadata subsystem, and wherein the second storage service node is configured to execute a second instance of the metadata subsystem, and wherein the commissioning comprises re-distributing the set of keys among storage service nodes that execute a respective instance of the metadata subsystem, including re-distributing the first range of keys between the first instance of the metadata subsystem and the second instance of the metadata subsystem.
19. The system of claim 13 , wherein before the second storage service node is activated, the first storage service node operates according to a current view that indicates that the first range of keys is associated with the first storage service node, and wherein after the second storage service node is activated, the first storage service node acquires a new view that indicates that the second range of keys is associated with the second storage service node;
wherein the at least one storage service node is configured to: inform the first storage service node and the second storage service node that the copying has been successfully completed, discard the current view, and establish the new view as the current view for the system, wherein establishment of the new view signifies completion of the commissioning of the second storage service node.
20. The system of claim 13 , wherein the at least one storage service node is further configured to control decommissioning of the second storage service node from the system without interruption of read requests from and write requests to any of the plurality of storage service nodes, wherein the decommissioning re-distributes ownership of keys within the second range of keys to at least one other storage service node within the system that is distinct from the second storage service node; and
wherein based on completion of the decommissioning, the second storage service node is not authorized to process any read requests and any write requests within the system.