Fast geo recovery method for elastic cloud storage
One embodiment is related to a method for remote replication recovery, comprising: determining that a damaged data chunk at a first zone of a cloud storage system is not recoverable locally; determining one or more data fragments of the damaged data chunk that are to be recovered with remote replication recovery based on data stored at a second zone and at a third zone of the cloud storage system, wherein the damaged data chunk comprises a plurality of data fragments; recovering the one or more data fragments of the damaged data chunk with remote replication recovery based on the data stored at the second zone and at the third zone of the cloud storage system; and repairing the damaged data chunk with the recovered data fragments at the first zone.
1. A method for remote replication recovery, comprising:
determining that a damaged data chunk at a first zone of a cloud storage system is not recoverable locally, wherein each data chunk corresponds to storage space of a predetermined fixed size and comprises a plurality of data fragments that are protected with one or more coding fragments generated based on erasure coding;
determining one or more data fragments of the damaged data chunk that are to be recovered with remote replication recovery based on data stored at a second zone and at a third zone of the cloud storage system;
recovering the one or more data fragments of the damaged data chunk with remote replication recovery based on the data stored at the second zone and at the third zone of the cloud storage system, wherein during the remote replication recovery the data fragments to be recovered are each identified within the respective data chunk with a fragment offset and a fragment size; and
subsequent to recovering the one or more data fragments of the damaged data chunk, repairing the damaged data chunk with the recovered data fragments at the first zone.
2. The method of claim 1 , wherein determining that the damaged data chunk at the first zone of the cloud storage system is not recoverable locally further comprises determining that the damaged data chunk is not recoverable using erasure coding based on data stored within the first zone.
3. The method of claim 1 , wherein the first, second, and third zones are geographically distributed.
4. The method of claim 1 , wherein the one or more data fragments that are to be recovered with remote replication recovery comprise all of invalid data fragments of the damaged data chunk.
5. The method of claim 1 , wherein the one or more data fragments that are to be recovered with remote replication recovery are determined so that remaining invalid data fragments not recovered with remote replication recovery are recoverable locally at the first zone with erasure coding.
6. The method of claim 5 , further comprising recovering the remaining invalid data fragments of the damaged data chunk locally at the first zone with erasure coding.
7. The method of claim 1 , wherein recovering the one or more data fragments of the damaged data chunk with remote replication recovery further comprises performing a bitwise exclusive or (XOR) operation on at least part of the data stored at the second zone and at the third zone of the cloud storage system.
8. The method of claim 1 , further comprising protecting the repaired data chunk with erasure coding at the first zone.
9. A non-transitory machine-readable medium having instructions stored therein which, when executed by a processor, cause the processor to perform testing operations, the operations comprising:
determining that a damaged data chunk at a first zone of a cloud storage system is not recoverable locally, wherein each data chunk corresponds to storage space of a predetermined fixed size and comprises a plurality of data fragments that are protected with one or more coding fragments generated based on erasure coding;
determining one or more data fragments of the damaged data chunk that are to be recovered with remote replication recovery based on data stored at a second zone and at a third zone of the cloud storage system;
recovering the one or more data fragments of the damaged data chunk with remote replication recovery based on the data stored at the second zone and at the third zone of the cloud storage system, wherein during the remote replication recovery the data fragments to be recovered are each identified within the respective data chunk with a fragment offset and a fragment size; and
subsequent to recovering the one or more data fragments of the damaged data chunk, repairing the damaged data chunk with the recovered data fragments at the first zone.
10. The non-transitory machine-readable medium of claim 9 , wherein determining that the damaged data chunk at the first zone of the cloud storage system is not recoverable locally further comprises determining that the damaged data chunk is not recoverable using erasure coding based on data stored within the first zone.
11. The non-transitory machine-readable medium of claim 9 , wherein the first, second, and third zones are geographically distributed.
12. The non-transitory machine-readable medium of claim 9 , wherein the one or more data fragments that are to be recovered with remote replication recovery comprise all of invalid data fragments of the damaged data chunk.
13. The non-transitory machine-readable medium of claim 9 , wherein the one or more data fragments that are to be recovered with remote replication recovery are determined so that remaining invalid data fragments not recovered with remote replication recovery are recoverable locally at the first zone with erasure coding.
14. The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise recovering the remaining invalid data fragments of the damaged data chunk locally at the first zone with erasure coding.
15. The non-transitory machine-readable medium of claim 9 , wherein recovering the one or more data fragments of the damaged data chunk with remote replication recovery further comprises performing a bitwise exclusive or (XOR) operation on at least part of the data stored at the second zone and at the third zone of the cloud storage system.
16. The non-transitory machine-readable medium of claim 9 , wherein the operations further comprise protecting the repaired data chunk with erasure coding at the first zone.
17. A data processing system, comprising:
a processor; and
a memory coupled to the processor storing instructions which, when executed by the processor, cause the processor to perform testing operations, the operations including
determining that a damaged data chunk at a first zone of a cloud storage system is not recoverable locally, wherein each data chunk corresponds to storage space of a predetermined fixed size and comprises a plurality of data fragments that are protected with one or more coding fragments generated based on erasure coding;
determining one or more data fragments of the damaged data chunk that are to be recovered with remote replication recovery based on data stored at a second zone and at a third zone of the cloud storage system;
recovering the one or more data fragments of the damaged data chunk with remote replication recovery based on the data stored at the second zone and at the third zone of the cloud storage system, wherein during the remote replication recovery the data fragments to be recovered are each identified within the respective data chunk with a fragment offset and a fragment size; and
subsequent to recovering the one or more data fragments of the damaged data chunk, repairing the damaged data chunk with the recovered data fragments at the first zone.
18. The data processing system of claim 17 , wherein determining that the damaged data chunk at the first zone of the cloud storage system is not recoverable locally further comprises determining that the damaged data chunk is not recoverable using erasure coding based on data stored within the first zone.
19. The data processing system of claim 17 , wherein the first, second, and third zones are geographically distributed.
20. The data processing system of claim 17 , wherein the one or more data fragments that are to be recovered with remote replication recovery comprise all of invalid data fragments of the damaged data chunk.
21. The data processing system of claim 17 , wherein the one or more data fragments that are to be recovered with remote replication recovery are determined so that remaining invalid data fragments not recovered with remote replication recovery are recoverable locally at the first zone with erasure coding.
22. The data processing system of claim 21 , wherein the operations further comprise recovering the remaining invalid data fragments of the damaged data chunk locally at the first zone with erasure coding.
23. The data processing system of claim 17 , wherein recovering the one or more data fragments of the damaged data chunk with remote replication recovery further comprises performing a bitwise exclusive or (XOR) operation on at least part of the data stored at the second zone and at the third zone of the cloud storage system.
24. The data processing system of claim 17 , wherein the operations further comprise protecting the repaired data chunk with erasure coding at the first zone.