Use of cluster-level redundancy within a cluster of a distributed storage management system to address node-level errors

1 . A method comprising:

managing, by an instance of a key-value (KV) store of a first node of a plurality of nodes of a cluster of a distributed storage system, storage of data blocks as values and corresponding block identifiers (IDs) as keys;

identifying a list of missing block IDs that are in use for one or more volumes associated with the first node but that are missing from the instance of the KV store by performing a data integrity check on the instance of the KV store; and

after identifying the list of missing block IDs, avoiding treating the first node as failed and instead performing a resynchronization process by, for each block ID in the list of block IDs:

reading a data block corresponding to the block ID from a second node of the plurality of nodes that maintains a redundant data block corresponding to the block ID; and

restoring the block ID within the KV store by writing the redundant data block to the first node.

2 . The method of claim 1 , wherein the second node is one of multiple nodes of the plurality of nodes that maintains the redundant data block.

3 . The method of claim 1 , wherein the data integrity check is separate from and performed independently of a garbage collection process.

4 . The method of claim 1 , wherein the data integrity check is integrated with and performed as part of a garbage collection process.

5 . The method of claim 1 , wherein the block IDs in the list of missing block IDs are associated with one or more bins of a plurality of bins each representing a subset of block IDs in a range of block IDs, wherein each of the one or more bins are assigned to one or more block services running on the first node and a respective redundant block service within the cluster, and wherein the one or more block services and the respective redundant block service are operable to store and retrieve data blocks associated with the assigned bins based on their respective block IDs.

6 . The method of claim 5 , further comprising identifying the redundant block service based on a bin of the one or more bins with which the block ID is associated, wherein an association between a given block ID and a given bin is determined based on a subset of bits of the given block ID.

7 . A non-transitory machine readable medium storing instructions, which when executed by one or more hardware processors of a cluster of a plurality of nodes representing a distributed storage system, cause the distributed storage system to:

identify a list of missing block identifiers (IDs) that are in use for one or more volumes associated with a first node of the plurality of nodes but that are missing from an instance of a key-value (KV) store of the first node by performing a data integrity check on the instance of the KV store; and

after identifying the list of missing block IDs, avoiding treating the first node as failed and instead performing a resynchronization process by, for each block ID in the list of block IDs:

reading a data block corresponding to the block ID from a second node of the plurality of nodes that maintains a redundant data block corresponding to the block ID; and

restoring the block ID within the KV store by writing the redundant data block to the first node.

8 . The non-transitory machine readable medium of claim 7 , wherein the data integrity check involves performing a walkthrough of mapped logical block addresses to identify block IDs that are currently in use for the one or more volumes.

9 . The non-transitory machine readable medium of claim 7 , wherein the second node is one of multiple nodes of the plurality of nodes that maintains the redundant data block.

10 . The non-transitory machine readable medium of claim 7 , wherein the data integrity check is separate from and performed independently of a garbage collection process.

11 . The non-transitory machine readable medium of claim 7 , wherein the data integrity check is integrated with and performed as part of a garbage collection process.

12 . The non-transitory machine readable medium of claim 7 , wherein the block IDs in the list of missing block IDs are associated with one or more bins of a plurality of bins each representing a subset of block IDs in a range of block IDs, wherein each of the one or more bins are assigned to one or more block services running on the first node and a respective redundant block service within the cluster, and wherein the one or more block services and the respective redundant block service are operable to store and retrieve data blocks associated with the assigned bins based on their respective block IDs.

13 . The non-transitory machine readable medium of claim 12 , wherein the instructions further cause the distributed storage system to identify the redundant block service based on a bin of the one or more bins with which the block ID is associated, wherein an association between a given block ID and a given bin is determined based on a subset of bits of the given block ID.

14 . A distributed storage system comprising:

a plurality of nodes operating as a cluster and including one or more hardware processors and instructions that when executed by the one or more hardware processors cause the distributed storage system to:

manage, within an instance of a key-value (KV) store of a first node of the plurality of nodes, storage of data blocks as values and corresponding block identifiers (IDs) as keys; and

avoid treating the first node as failed and instead performing a resynchronization process by, for each block ID in a list of missing block IDs that are in use for one or more volumes associated with the first node but that are missing from the instance of the KV store:

reading a data block corresponding to the block ID from a second node of the plurality of nodes that maintains a redundant data block corresponding to the block ID; and

restoring the block ID within the KV store by writing the redundant data block to the first node.

15 . The distributed storage system of claim 14 , wherein the second node is one of multiple nodes of the plurality of nodes that maintains the redundant data block.

16 . The distributed storage system of claim 14 , wherein the list of missing block IDs is identified by performing a data integrity check on the instance of the KV store.

17 . The distributed storage system of claim 16 , wherein the data integrity check is separate from and performed independently of a garbage collection process.

18 . The distributed storage system of claim 16 , wherein the data integrity check is integrated with and performed as part of a garbage collection process.

19 . The distributed storage system of claim 14 , wherein the block IDs in the list of missing block IDs are associated with one or more bins of a plurality of bins each representing a subset of block IDs in a range of block IDs, wherein each of the one or more bins are assigned to one or more block services running on the first node and a respective redundant block service within the cluster, and wherein the one or more block services and the respective redundant block service are operable to store and retrieve data blocks associated with the assigned bins based on their respective block IDs.

20 . The distributed storage system of claim 19 , wherein the instructions further cause the distributed storage system to identify the redundant block service based on a bin of the one or more bins with which the block ID is associated, wherein an association between a given block ID and a given bin is determined based on a subset of bits of the given block ID.

21 . The distributed storage system of claim 16 , wherein the data integrity check involves performing a walkthrough of mapped logical block addresses to identify block IDs that are currently in use for the one or more volumes.