Virtual chunk service based data recovery in a distributed data storage system
Technology is disclosed for storing data in a distributed storage system using a virtual chunk service (VCS). In the VCS based storage technique, a storage node (“node”) is split into multiple VCSs and each of the VCSs can be assigned a unique ID in the distributed storage. A set of VCSs from a set of nodes form a storage group, which also can be assigned a unique ID in the distributed storage. When a data object is received for storage, a storage group is identified for the data object, the data object is encoded to generate multiple fragments and each fragment is stored in a VCS of the identified storage group. The data recovery process is made more efficient by using metadata, e.g., VCS to storage node mapping, storage group to VCS mapping, VCS to objects mapping, which eliminates resource intensive read and write operations during recovery.
1. A computer-implemented method comprising:
determining that a first data storage node of a distributed storage system having multiple data storage nodes has failed or is about to fail;
based on determining that the first data storage node has failed or is about to fail,
identifying a first set of virtual chunk spaces assigned to the first data storage node based on first metadata that indicate mappings between data storage nodes and virtual chunk spaces, wherein a plurality of virtual chunk spaces across the distributed storage system includes the first set of virtual chunk spaces;
reassigning the first set of virtual chunk spaces to a second data storage node of the distributed storage system;
identifying a first set of data objects of which fragments had been stored to the first set of virtual chunk spaces; and
after reassignment of the first set of virtual chunk spaces to the second data storage node and after regeneration of a first set of fragments of the first set of data objects that were stored to the first set of virtual chunk spaces when assigned to the first data storage node, storing the first set of fragments to the first set of virtual chunk spaces.
2. The computer-implemented method of claim 1 , wherein reassigning the first set of virtual chunk spaces to the second data storage node comprises:
updating the mappings in the first metadata to indicate that the first set of virtual chunk spaces is assigned to the second data storage node instead of the first data storage node.
3. The computer-implemented method of claim 1 further comprising:
identifying a first set of erasure coding groups associated with the first set of virtual chunk spaces based on second metadata that indicate associations of erasure coding groups with virtual chunk spaces, wherein the first data storage node is assigned to no more than one of the first set of virtual chunk spaces from each of the first set of erasure coding groups,
wherein identifying the first set of data objects of which fragments have been stored to the first set of virtual chunk spaces is based on third metadata that indicates mappings of data objects to erasure coding groups.
4. The computer-implemented method of claim 3 further comprising:
based on a client request for retrieving a first data object of the first set of data objects from the distributed storage system, the request including a storage group identification (ID) of a first of the first set of erasure coding groups with which the first data object is associated:
identifying a second set of the virtual chunk spaces associated with the first erasure coding group according to the second metadata and retrieving fragments of the first data object from the second set of virtual chunk spaces, wherein the second set of virtual chunk spaces includes only one of the first set of virtual chunk spaces; and
returning the first data object to the requesting client after generating the first data object from the retrieved fragments.
5. The computer-implemented method of claim 1 , wherein identifying the first set of data objects of which fragments have been stored to the first set of virtual chunk spaces is based on third metadata that indicates mappings of virtual chunk spaces to data objects.
6. A non-transitory computer-readable storage medium having computer-executable instructions, comprising instructions to:
determine that a first data storage node of a distributed storage system having multiple data storage nodes has failed or is about to fail;
based on a determination that the first data storage node has failed or is about to fail,
identify a first set of virtual chunk spaces assigned to the first data storage node based on first metadata that indicate mappings between data storage nodes and virtual chunk spaces, wherein a plurality of virtual chunk spaces across the distributed storage system includes the first set of virtual chunk spaces;
reassign the first set of virtual chunk spaces to a second data storage node of the distributed storage system;
identify a first set of data objects of which fragments had been stored to the first set of virtual chunk spaces; and
after reassignment of the first set of virtual chunk spaces to the second data storage node and after regeneration of a first set of fragments of the first set of data objects that were stored to the first set of virtual chunk spaces when assigned to the first data storage node, store the first set of fragments to the first set of virtual chunk spaces.
7. The non-transitory computer-readable storage medium of claim 6 , wherein the instructions to reassign the first set of virtual chunk spaces to the second data storage node comprise instructions to:
update the mappings in the first metadata to indicate that the first set of virtual chunk spaces is assigned to the second data storage node instead of the first data storage node.
8. The non-transitory computer-readable storage medium of claim 6 , wherein the computer-executable instructions further comprise instructions to:
identify a first set of erasure coding groups associated with the first set of virtual chunk spaces based on second metadata that indicate associations of erasure coding groups with virtual chunk spaces, wherein the first data storage node is assigned to no more than one of the first set of virtual chunk spaces from each of the first set of erasure coding groups,
wherein the instructions to identify the first set of data objects of which fragments have been stored to the first set of virtual chunk spaces comprise instructions to identify the first set of data objects based on third metadata that indicate mappings of data objects to erasure coding groups.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the computer-executable instructions further comprise instructions to:
based on a request from a client computer for retrieving a first data object of the first set of data objects from the distributed storage system, the request including a storage group identification (ID) of a first of the first set of erasure coding groups with which the first data object is associated:
identify a second set of virtual chunk spaces associated with the first erasure coding group according to the second metadata and retrieve fragments of the first data object from the second set of virtual chunk spaces, wherein the second set of virtual chunk spaces includes only one of the first set of virtual chunk spaces; and
return the first data object to the client computer after generation of the first data object from the retrieved fragments.
10. The non-transitory computer-readable storage medium of claim 6 , wherein the instructions to identify the first set of data objects of which fragments have been stored to the first set of virtual chunk spaces comprise instructions to identify the first set of data objects based on third metadata that indicate mappings of virtual chunk spaces to data objects.
11. A system comprising:
a processor;
a non-transitory computer-readable medium comprising instructions executable by the processor to cause the system to,
determine that a first data storage node of a distributed storage system having multiple data storage nodes has failed or is about to fail;
based on a determination that the first data storage node has failed or is about to fail,
identify a first set of virtual chunk spaces assigned to the first data storage node based on first metadata that indicate mappings between data storage nodes and virtual chunk spaces, wherein a plurality of virtual chunk spaces across the distributed storage system includes the first set of virtual chunk spaces;
reassign the first set of virtual chunk spaces to a second data storage node of the distributed storage system;
identify a first set of data objects of which fragments had been stored to the first set of virtual chunk spaces; and
after reassignment of the first set of virtual chunk spaces to the second data storage node and after regeneration of a first set of fragments of the first set of data objects that were stored to the first set of virtual chunk spaces when assigned to the first data storage node, store the first set of fragments to the first set of virtual chunk spaces.
12. The system of claim 11 , wherein the instructions to reassign the first set of virtual chunk spaces to the second data storage node comprise instructions executable by the processor to cause the system to:
update the mappings in the first metadata to indicate that the first set of virtual chunk spaces is assigned to the second data storage node instead of the first data storage node.
13. The system of claim 11 , wherein the non-transitory computer-readable medium further comprises instructions executable by the processor to cause the system to:
identify a first set of erasure coding groups associated with the first set of virtual chunk spaces based on second metadata that indicate associations of erasure coding groups with virtual chunk spaces, wherein the first data storage node is assigned to no more than one of the first set of virtual chunk spaces from each of the first set of erasure coding groups,
wherein the instructions to identify the first set of data objects of which fragments have been stored to the first set of virtual chunk spaces comprise instructions to identify the first set of data objects based on third metadata that indicate mappings of data objects to erasure coding groups.
14. The system of claim 13 , wherein the non-transitory computer-readable medium further comprises instructions executable by the processor to cause the system to:
based on a request from a client computer for retrieving a first data object of the first set of data objects from the distributed storage system, the request including a storage group identification (ID) of a first of the first set of erasure coding groups with which the first data object is associated:
identify a second set of virtual chunk spaces associated with the first erasure coding group according to the second metadata and retrieve fragments of the first data object from the second set of virtual chunk spaces, wherein the second set of virtual chunk spaces includes only one of the first set of virtual chunk spaces; and
return the first data object to the client computer after generation of the first data object from the retrieved fragments.
15. The system of claim 11 , wherein the instructions to identify the first set of data objects of which fragments have been stored to the first set of virtual chunk spaces comprise instructions to identify the first set of data objects based on third metadata that indicate mappings of virtual chunk spaces to data objects.