Distributed site data recovery in a geographically distributed data storage environment
The described technology is generally directed towards recovering a chunk (or similar block of data) when the chunk is erasure coded into fragments, and recovery fragments need to be obtained from geographically distributed sites. The recovery fragments needed to perform recovery of a chunk are determined, and assigned to the geographically distributed sites as subtasks. Each site that receives a subtask from the requesting site obtains XOR-related fragments needed to produce the recovery fragment, performs the XOR operations on the XOR-related fragments to produce the recovery fragment, and returns the recovery fragment to the requesting site. When finished, a site receives another subtask until no subtasks remain, such that the fastest site or sites receive the most subtasks. The requesting site recovers the chunk from the received recovery fragments. The shared participation in the chunk recovery among the distributed sites provides for efficient distribution of the recovery-related resources and work.
1. A system, comprising:
a processor, and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
determining, at a local site of a group of sites, that a chunk of the local site is to be recovered;
determining chunk recovery fragments that are part of recovery of the chunk;
assigning respective subtasks, respectively corresponding to obtaining the chunk recovery fragments, to respective sites of the group of sites;
in response to receiving an error corresponding to assignment of a subtask of the respective subtasks, reassigning the subtask of the respective subtasks;
in response to the assigning of the respective subtasks, receiving the chunk recovery fragments; and
in response to the receiving of the chunk recovery fragments, recovering the chunk from the chunk recovery fragments.
2. The system of claim 1 , wherein the assigning of the respective subtasks comprises assigning a subtask to the local site.
3. The system of claim 1 , wherein the assigning of the respective subtasks comprises assigning the respective subtasks corresponding to obtaining a subgroup of the chunk recovery fragments, and wherein the subgroup comprises the chunk recovery fragments that are part of the recovery of the chunk minus at least one recovery fragment determined to be already present on the local site.
4. The system of claim 3 , wherein the subgroup is determined from a minimum number of chunk recovery fragments determined to be needed to perform the recovery of the chunk based on an erasure coding scheme minus the at least one recovery fragment determined to be already present on the local site.
5. The system of claim 1 , wherein a subtask of the respective subtasks corresponds to obtaining exclusive OR (XOR)-related recovery fragments, and XOR-ing the XOR-related recovery fragments into a chunk recovery fragment.
6. The system of claim 1 , wherein the chunk recovery fragments are data fragments corresponding to an erasure coding scheme, and wherein the recovering the chunk from the chunk recovery fragments comprises encoding the data fragments into coding fragments according to the erasure coding scheme, and storing the data fragments and the coding fragments.
7. The system of claim 1 , wherein the chunk recovery fragments comprise data fragments and a coding fragment corresponding to an erasure coding scheme, and wherein the recovering the chunk from the chunk recovery fragments comprises decoding a data fragment via the coding fragments, encoding the data fragments according to the erasure coding scheme into coding fragments, and storing the data fragments and the coding fragments.
8. A method comprising:
receiving, by a system comprising a processor at a first site of a group of geographically distributed sites, a request comprising a subtask to return a chunk recovery fragment to a second site of the group;
in response to detecting an error corresponding to an assignment of the subtask comprised in the request, reassigning, by the system, the subtask;
obtaining, by the system, a group of exclusive OR (XOR)-related recovery fragments corresponding to the chunk recovery fragment;
performing, by the system, logical XOR operations on the group of XOR-related recovery fragments to produce the chunk recovery fragment; and
returning, by the system, the chunk recovery fragment in response to the request.
9. The method of claim 8 , wherein the obtaining the group of XOR-related recovery fragments comprises obtaining an XOR-related recovery fragment of the group of XOR-related recovery fragments from the second site.
10. The method of claim 8 , wherein the obtaining the group of XOR-related recovery fragments comprises obtaining an XOR-related recovery fragment of the group of XOR-related recovery fragments from a third site of the group.
11. The method of claim 8 , wherein the first site comprises local node storage, and wherein the obtaining the group of XOR-related recovery fragments comprises obtaining an XOR-related recovery fragment from the local node storage.
12. The method of claim 11 , wherein the obtaining the XOR-related recovery fragment from the local node storage comprises performing a local recovery operation.
13. The method of claim 8 , wherein the request is a first request comprising a first subtask to return a first chunk recovery fragment, wherein the group of XOR-related recovery fragments comprises a first group of XOR-related recovery fragments, and further comprising receiving, by the system at the first site, a second request comprising a second subtask to return a second chunk recovery fragment to the second site of the group, obtaining, by the system, a second group of XOR-related recovery fragments corresponding to the second chunk recovery fragment, performing, by the system, logical XOR operations on the second group of XOR-related recovery fragments to produce the second chunk recovery fragment, and returning, by the system, the second chunk recovery fragment in response to the second request.
14. The method of claim 8 , wherein the assigning of the subtasks comprises assigning, by the system, the subtask to a local site.
15. A non-transitory machine-readable storage medium, comprising executable instructions that, when executed by a processor of a local site of a group of geographically distributed sites, facilitate performance of operations, the operations comprising:
obtaining recovery fragments for recovery of a data chunk, comprising directing respective sites of the group to provide respective chunk recovery fragments, wherein the obtaining the recovery fragments comprises assigning respective subtasks, corresponding to obtaining the recovery fragments, to a subgroup of the respective chunk recovery fragments, and wherein, in response to receiving an error corresponding to assignment of a subtask of the respective subtasks, the subtask of the respective subtasks is reassigned; and
recovering the chunk from the respective chunk recovery fragments, comprising encoding the respective chunk recovery fragments according to an erasure coding scheme.
16. The non-transitory machine-readable storage medium of claim 15 , wherein the chunk recovery fragments comprise data fragments corresponding to an erasure coding process, and wherein the recovering the chunk from the chunk recovery fragments comprises encoding the data fragments into coding fragments according to the erasure coding process, and storing the data fragments and the coding fragments.
17. The non-transitory machine-readable storage medium of claim 15 , wherein the operations further comprise determining a subgroup of needed chunk recovery fragments for the recovering of the chunk, the subgroup being determined based on a minimum number of chunk recovery fragments determined to be needed for recovery and any chunk recovery fragments present at the local site, and wherein the subgroup of the respective chunk recovery fragments corresponds to the subgroup of the needed chunk recovery fragments.
18. The non-transitory machine-readable storage medium of claim 15 , wherein the directing the respective sites of the group to return the respective chunk recovery fragments comprises directing the local site to provide a first chunk recovery fragment, and directing a remote site to provide a second chunk recovery fragment.
19. The non-transitory machine-readable storage medium of claim 18 , wherein the directing the respective sites of the group to return the respective chunk recovery fragments comprises directing the remote site to provide a third chunk recovery fragment.
20. The non-transitory machine-readable storage medium of claim 15 , wherein the directing the respective sites of the group to return the respective chunk recovery fragments comprises directing the local site to provide a first chunk recovery fragment, and wherein the operations further comprise, obtaining, at the local site, exclusive OR (XOR)-related fragments that are related to the first chunk recovery fragment by XOR operations, comprising obtaining at least one of the XOR-related fragments from a remote site, and performing XOR operations on the XOR-related fragments to produce the first chunk recovery fragment.