Healing failed erasure-coded write attempts in a distributed data storage system configured with fewer storage nodes than data plus parity fragments
A distributed data storage system using erasure coding (EC) provides advantages of EC data storage while retaining high resiliency for EC data storage architectures having fewer data storage nodes than the number of EC data-plus-parity fragments. To ameliorate the effects of certain storage node outages or fatal disk failures, incoming data is temporarily replicated so that read and write operations can continue from/to the storage system. The system automatically heals failed EC write attempts in a manner transparent to users and/or applications: when all storage nodes are operational, the distributed data storage system automatically converts the temporarily replicated data to EC storage and reclaims storage space previously used by the temporarily replicated data. Individual hardware failures are healed through migration techniques that reconstruct and re-fragment data blocks according to the governing EC scheme. An illustrative embodiment is a three-node data storage system using EC 4+2.
1. A data storage appliance using an erasure coding scheme that generates a count of N erasure-coded fragments including data fragments and parity fragments of a data block, the data storage appliance comprising:
a plurality of storage service nodes fewer than the count of N,
wherein each storage service node comprises a plurality of physical data storage resources,
wherein a first one of the plurality of storage service nodes is configured to:
based on a failure to successfully store each of N erasure-coded fragments of a first data block within the data storage appliance, cause at least two instances of the first data block, in unfragmented form, to be stored successfully within the data storage appliance,
wherein each of the at least two instances of the first data block has been stored at a physical data storage resource, which is separate from other physical data storage resources that host others of the at least two instances of the first data block in unfragmented form;
based on detecting that all of the plurality of storage service nodes are in service after the first data block was successfully written in unfragmented form:
read an instance the first data block in unfragmented form,
generate N erasure-coded fragments of the first data block by applying the erasure coding scheme,
receive confirmations that each of the N erasure-coded fragments of the first data block was successfully stored at a physical data storage resource among the plurality of physical data storage resources,
wherein only one instance of each of the N erasure-coded fragments of the first data block is stored in the data storage appliance,
wherein the N erasure-coded fragments of the first data block are distributed among the plurality of storage service nodes in the data storage appliance, and
wherein each one of the N erasure-coded fragments of the first data block is stored in a physical data storage resource that is separate from other physical data storage resources hosting others of the N erasure-coded fragments of the first data block.
2. The data storage appliance of claim 1 , wherein the first one of the plurality of storage service nodes is further configured to:
after generating the N erasure-coded fragments of the first data block, cause each of the N erasure-coded fragments to be stored at a physical data storage resource that is separate from other physical data storage resources hosting others of the N erasure-coded fragments of the first data block.
3. The data storage appliance of claim 1 , wherein the first one of the plurality of storage service nodes is further configured to:
after the N erasure-coded fragments of the first data block are successfully stored within the data storage appliance, cause all instances of the first data block, in unfragmented form, to be deleted from the plurality of storage service nodes.
4. The data storage appliance of claim 3 , wherein the first one of the plurality of storage service nodes is further configured to: wait a pre-determined interval of time before causing all instances of the first data block, in unfragmented form, to be deleted from the plurality of storage service nodes.
5. The data storage appliance of claim 1 , wherein the first one of the plurality of storage service nodes is further configured to:
cause a replication-factor virtual disk to be removed from the data storage appliance after all unfragmented data blocks in the replication-factor virtual disk, including the first data block in unfragmented form, have been deleted.
6. The data storage appliance of claim 1 , wherein the first one of the plurality of storage service nodes is further configured to: execute a healing process that identifies one or more data blocks stored in unfragmented form in a replication-factor virtual disk, including the first data block;
wherein the first one of the plurality of storage service nodes uses the healing process to perform the read, the generate, and the receive, and to:
after the N erasure-coded fragments of the first data block are generated, cause each of the N erasure-coded fragments to be stored within the data storage appliance, and
after the N erasure-coded fragments of the first data block are successfully stored within the data storage appliance, cause all instances of the first data block, in unfragmented form, to be deleted from the data storage appliance.
7. The data storage appliance of claim 6 , wherein the healing process further performs:
cause a replication-factor virtual disk to be removed from the data storage appliance after all unfragmented data blocks in the replication-factor virtual disk, including the first data block in unfragmented form, have been deleted.
8. The data storage appliance of claim 1 , wherein the first one of the plurality of storage service nodes is further configured to:
cause the at least two instances of the first data block, in unfragmented form, to be stored in a replication-factor virtual disk, which is associated with an erasure-coded virtual disk for storing the N erasure-coded fragments of the first data block;
wait a pre-determined amount of time before a healing process that executes at the first one of the plurality of storage service nodes identifies one or more data blocks stored in unfragmented form in the replication-factor virtual disk, including the first data block;
to each of the one or more data blocks, apply the erasure coding scheme to generate N respective erasure-coded fragments; and
cause the N respective erasure-coded fragments to be stored to the erasure-coded virtual disk, which is distributed among the plurality of storage service nodes.
9. The data storage appliance of claim 1 , wherein the plurality of storage service nodes is three storage service nodes, and wherein each of the three storage service nodes comprises at least three physical data storage resources; and
wherein the erasure coding scheme generates four data fragments and two parity fragments, and wherein the count of N erasure-coded fragments is six.
10. The data storage appliance of claim 1 , wherein each physical data storage resource is configured as a distinct storage pool.
11. A computer-implemented method based on an erasure coding scheme that generates a count of N erasure-coded fragments including data fragments and parity fragments of a data block, the method comprising:
by a first storage service node among a plurality of storage service nodes of a distributed data storage system, wherein the plurality of storage service nodes is fewer than the count of N, and
wherein each storage service node comprises one or more hardware processors and a plurality of physical data storage resources:
based on a failure to successfully store each of N erasure-coded fragments of a first data block within the distributed data storage system, causing at least two instances of the first data block, in unfragmented form, to be stored successfully within the distributed data storage system,
wherein each of the at least two instances of the first data block is stored at a physical data storage resource, which is separate from other physical data storage resources that host others of the at least two instances of the first data block in unfragmented form;
based on detecting that all of the plurality of storage service nodes are in service after the first data block was successfully written in unfragmented form:
reading an instance of the first data block in unfragmented form,
generating N erasure-coded fragments of the first data block by applying the erasure coding scheme,
causing each of the N erasure-coded fragments to be stored at a physical data storage resource that is separate from other physical data storage resources hosting others of the N erasure-coded fragments of the first data block, and
receiving confirmations that each of the N erasure-coded fragments of the first data block was successfully stored,
wherein only one instance of each of the N erasure-coded fragments of the first data block is stored in the distributed data storage system,
wherein the N erasure-coded fragments of the first data block are distributed among the plurality of storage service nodes in the distributed data storage system.
12. The method of claim 11 further comprising:
after the N erasure-coded fragments of the first data block are successfully stored within the distributed data storage system, causing all instances of the first data block, in unfragmented form, to be deleted from the plurality of storage service nodes.
13. The method of claim 11 further comprising: waiting a pre-determined interval of time before causing all instances of the first data block, in unfragmented form, to be deleted from the plurality of storage service nodes.
14. The method of claim 11 further comprising: causing a replication-factor virtual disk to be removed from the distributed data storage system after all unfragmented data blocks in the replication-factor virtual disk, including the first data block in unfragmented form, have been deleted.
15. The method of claim 11 , wherein the first storage service node executes a healing process that detects that all of the plurality of storage service nodes are in service and performs the reading, the generating, the causing, and the receiving.
16. The method of claim 11 further comprising:
executing a healing process that identifies one or more data blocks stored in unfragmented form in a replication-factor virtual disk of the distributed data storage system, including identifying the first data block;
wherein the healing process performs the reading, the generating, the causing, and the receiving, and further performs:
after the N erasure-coded fragments of the first data block are generated, causing each of the N erasure-coded fragments to be stored within the distributed data storage system, and
after the N erasure-coded fragments of the first data block are successfully stored within the distributed data storage system, causing all instances of the first data block, in unfragmented form, to be deleted from the distributed data storage system.
17. The method of claim 16 , wherein the healing process further performs:
causing a replication-factor virtual disk to be removed from the distributed data storage system after all unfragmented data blocks in the replication-factor virtual disk, including the first data block in unfragmented form, have been deleted.
18. The method of claim 11 further comprising:
causing the at least two instances of the first data block, in unfragmented form, to be stored in a replication-factor virtual disk of the distributed data storage system, wherein the replication-factor virtual disk is associated with an erasure-coded virtual disk which is configured in the distributed data storage system for storing the N erasure-coded fragments of the first data block;
waiting a pre-determined amount of time before identifying one or more data blocks stored in unfragmented form in the replication-factor virtual disk, including the first data block;
to each of the one or more data blocks, applying the erasure coding scheme to generate N respective erasure-coded fragments; and
causing the N respective erasure-coded fragments to be stored into the erasure-coded virtual disk, which is distributed among the plurality of storage service nodes.
19. The method of claim 11 , wherein three storage service nodes form the plurality of storage service nodes, and wherein each of the three storage service nodes comprises at least three physical data storage resources; and
wherein the erasure coding scheme generates four data fragments and two parity fragments, and wherein the count of N erasure-coded fragments is six.
20. The method of claim 11 , wherein the failure to successfully store each of the N erasure-coded fragments of the first data block within the distributed data storage system is based on one or more of:
(a) fewer than all of the plurality of storage service nodes being available, and
(b) receiving fewer than N confirmations that each of the N erasure-coded fragments has been successfully stored at a physical data storage resource, and
(c) receiving fewer than a replication factor of confirmations that metadata for all of the N erasure-coded fragments has been successfully stored within the distributed data storage system.