Migrating erasure-coded data fragments within a growing distributed data storage system with at least data plus parity nodes
A distributed data storage system that employs erasure coding grows from fewer than data-plus-parity (D+P) storage service nodes to at least D+P nodes. The system detects an increase in the number of available storage service nodes, i.e., at least D+P; analyzes how storage for each virtual disk is distributed among storage containers in the existing (pre-growth) nodes; identifies containers that are co-hosted on the same node; and, on a node-by-node basis, migrates data fragments from a co-hosted storage container to a corresponding new container that is configured on another node. For each virtual disk in the illustrative system, the migration causes containers, including the erasure-coded data fragments they host, to be re-distributed so that the containers for a virtual disk are NOT doubled up or co-hosted on the same node. The disclosed computer-implemented process for re-distributing erasure-coded data fragments operates organically, without requiring the system to restart or reboot.
1 . A system comprising:
a first plurality of storage service nodes, wherein physical data storage resources are configured among the first plurality of storage service nodes; and
wherein a first storage service node among the first plurality of storage service nodes is configured to:
receive a write request comprising a first data block in unfragmented form, wherein the write request indicates that the first data block is to be written to a virtual disk configured in the system;
determine that the first data block is to be stored according to an erasure-coding scheme, which generates a count of D data fragments and a count of P parity fragments that total N erasure-coded fragments of the first data block;
based on determining that the first plurality of storage service nodes is fewer than N, cause the N erasure-coded fragments to be written at one or more storage service nodes among the first plurality of storage service nodes, wherein only one instance of each of the N erasure-coded fragments is stored among the first plurality of storage service nodes in the system, and wherein the first storage service node comprises a first one of the N erasure-coded fragments and further comprises a second one of the N erasure-coded fragments;
after the N erasure-coded fragments have been stored among the first plurality of storage service nodes that is fewer than N, detect that more storage service nodes are currently operational in the system;
based on determining that at least N storage service nodes are currently operational in the system, initiate a migration that moves the second one of the N erasure-coded fragments from the first storage service node to a second storage service node, wherein after the migration is complete, each of the N erasure-coded fragments is stored on a distinct storage service node in the system.
2 . The system of claim 1 , wherein the migration re-distributes storage containers of the virtual disk, including erasure-coded data fragments hosted therein, such that the storage containers of the virtual disk are not co-hosted on a same storage service node after the migration is complete.
3 . The system of claim 1 , wherein at least one of the first plurality of storage service nodes is configured to, responsive to the first storage service node initiating the migration, identify, among the more storage service nodes, the second storage service node that is suitable for the migration.
4 . The system of claim 1 , wherein a data storage subsystem that executes at the first storage service node is configured to: based on determining that the first storage service node comprises two storage containers of the virtual disk, request a metadata subsystem to identify another storage service node that is suitable to host one of the two storage containers;
wherein the metadata subsystem, which executes at one of the first plurality of storage service nodes, is configured to identify, among the more storage service nodes, a second storage service node that does not comprise any storage containers of the virtual disk and is further configured to indicate the second storage service node to the data storage subsystem for the migration.
5 . The system of claim 1 , wherein the first storage service node is further configured to:
after the migration is complete, receive a second write request comprising a second data block in unfragmented form, wherein the second write request indicates that the second data block is to be written to the virtual disk;
based on a configuration of the virtual disk programmed into the system, determine that the second data block is to be stored according to the erasure-coding scheme, which generates N erasure-coded fragments of the second data block; and
based on determining that at least N storage service nodes are currently operational in the system, cause the N erasure-coded fragments of the second data block to be written at N distinct storage service nodes among the at least N storage service nodes that are currently operational in the system, wherein only one instance of each of the N erasure-coded fragments of the second data block is stored at any one storage service node.
6 . The system of claim 1 , wherein the first storage service node is further configured to:
receive a read request for the first data block;
determine that the first data block is stored in the system as the N erasure-coded fragments of the first data block, which consist of D data fragments and P parity fragments;
obtain D erasure-coded fragments from among the N erasure-coded fragments of the first data block, wherein the count of D is sufficient to reconstruct the first data block according to the erasure-coding scheme;
reconstruct the first data block in unfragmented form from the D erasure-coded fragments; and
transmit the first data block in unfragmented form in response to the read request.
7 . The system of claim 1 , wherein the virtual disk comprises storage containers that are distributed among the first plurality of storage service nodes, and wherein the migration causes a first storage container configured at the first storage service node to move to the second storage service node, wherein after the migration is complete, none of the storage containers of the virtual disk are co-located at a same storage service node.
8 . The system of claim 1 , wherein a data storage subsystem that executes at the first storage service node is configured to initiate the migration.
9 . The system of claim 8 , wherein a metadata subsystem that executes at one of the first plurality of storage service nodes is configured to identify the second storage service node as a destination for the migration.
10 . The system of claim 1 , wherein the migration does not comprise a reboot of the system.
11 . The system of claim 1 , wherein the migration does not comprise a restart of the system.
12 . A system comprising:
a first plurality of storage service nodes, wherein physical data storage resources are configured among the first plurality of storage service nodes; and
wherein a first storage service node among the first plurality of storage service nodes is configured to:
receive a write request comprising a first data block in unfragmented form, wherein the write request indicates that the first data block is to be written to a virtual disk configured in the system, wherein the virtual disk comprises storage containers that are distributed among the first plurality of storage service nodes;
determine that the first data block is to be stored according to an erasure-coding scheme, which generates a count of N erasure-coded fragments of the first data block, wherein N consists of D data fragments and P parity fragments;
based on determining that the first plurality of storage service nodes is fewer than N, cause the N erasure-coded fragments to be written at one or more storage service nodes among the first plurality of storage service nodes, wherein only one instance of each of the N erasure-coded fragments is stored among the first plurality of storage service nodes in the system, and wherein a first storage container of the virtual disk configured at the first storage service node comprises a first one of the N erasure-coded fragments, and wherein a second storage container of the virtual disk also configured at the first storage service node comprises a second one of the N erasure-coded fragments;
after the N erasure-coded fragments have been stored among the first plurality of storage service nodes that is fewer than N, based on determining that at least N storage service nodes are currently operational in the system, initiate a migration that moves the second storage container of the virtual disk, including the second one of the N erasure-coded fragments, to a second storage service node that lacks any storage container of the virtual disk; and
wherein after the migration is complete, each of the N erasure-coded fragments is stored on a distinct storage service node in the system.
13 . The system of claim 12 , wherein after the migration is complete, none of the storage containers of the virtual disk are co-located at a same storage service node.
14 . The system of claim 12 , wherein the migration re-distributes the storage containers of the virtual disk, including erasure-coded data fragments hosted therein, such that the storage containers of the virtual disk are not co-hosted on a same storage service node after the migration is complete.
15 . The system of claim 12 , wherein at least one of the first plurality of storage service nodes is configured to, responsive to the first storage service node initiating the migration, identify the second storage service node that is suitable for the migration.
16 . The system of claim 12 , wherein a data storage subsystem that executes at the first storage service node is configured to: based on determining that the first storage service node comprises two storage containers of the virtual disk, request a metadata subsystem to identify another storage service node that is suitable to host one of the two storage containers;
wherein the metadata subsystem, which executes at one of the first plurality of storage service nodes, is configured to identify the second storage service node that does not comprise any storage containers of the virtual disk, and is further configured to indicate the second storage service node to the data storage subsystem for the migration.
17 . The system of claim 12 , wherein the first storage service node is further configured to:
after the migration is complete, receive a second write request comprising a second data block in unfragmented form, wherein the second write request indicates that the second data block is to be written to the virtual disk;
based on a configuration of the virtual disk programmed into the system, determine that the second data block is to be stored according to the erasure-coding scheme, which generates N erasure-coded fragments of the second data block; and
based on determining that at least N storage service nodes are currently operational in the system, cause the N erasure-coded fragments of the second data block to be written at N distinct storage service nodes among the at least N storage service nodes that are currently operational in the system, wherein only one instance of each of the N erasure-coded fragments of the second data block is stored at any one storage service node.
18 . The system of claim 12 , wherein the first storage service node is further configured to:
receive a read request for the first data block;
determine that the first data block is stored in the system as the N erasure-coded fragments of the first data block, which consist of D data fragments and P parity fragments;
obtain D erasure-coded fragments from among the N erasure-coded fragments of the first data block, wherein the count of D is sufficient to reconstruct the first data block according to the erasure-coding scheme;
reconstruct the first data block in unfragmented form from the D erasure-coded fragments; and
transmit the first data block in unfragmented form in response to the read request.
19 . The system of claim 12 , wherein a data storage subsystem that executes at the first storage service node is configured to initiate the migration, based on determining that more than one storage container of the virtual disk is configured at the first storage service node.
20 . The system of claim 19 , wherein a metadata subsystem that executes at one of the first plurality of storage service nodes is configured to identify the second storage service node as a destination for the migration.