System and method for capacity and network traffic efficient data protection on distributed storage system
One embodiment is related to a method for implementing a demoted chunk, comprising: replicating a first chunk from a first zone of a cloud storage system to a third zone of the cloud storage system; replicating a second chunk from a second zone of the cloud storage system to the third zone; creating an exclusive or (XOR) chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk; and marking the replicated first chunk and the replicated second chunk as demoted first chunk and demoted second chunk, respectively, at the third zone.
1. A method for implementing a demoted chunk, comprising:
replicating a first chunk from a first zone of a cloud storage system to a third zone of the cloud storage system;
replicating a second chunk from a second zone of the cloud storage system to the third zone;
creating an exclusive or (XOR) chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk; and
subsequent to the creation of the XOR chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk, marking the replicated first chunk and the replicated second chunk as locally demoted first chunk and locally demoted second chunk, respectively, at the third zone, wherein new data is allowed to be written at the third zone to space corresponding to any locally demoted chunk at the third zone, and wherein data of any locally demoted chunk at the third zone remains available unless and until the corresponding space at the third zone is overwritten.
2. The method of claim 1 , wherein the first, second, and third zones are located remotely from each other.
3. The method of claim 1 , wherein space occupied by the locally demoted first chunk and the locally demoted second chunk is categorized as available space at the third zone when the third zone determines storage space usage.
4. The method of claim 1 , wherein in response to a determination that the first chunk is lost at the first zone and the locally demoted first chunk at the third zone is available at the third zone, the first chunk is recovered by marking the locally demoted first chunk at the third zone as a regular chunk or copying the locally demoted first chunk at the third zone to the first zone.
5. The method of claim 1 , wherein in response to a determination that the first chunk is lost at the first zone and the locally demoted second chunk at the third zone is available at the third zone, the first chunk is recovered at the third zone based at least in part on the XOR chunk and the locally demoted second chunk at the third zone.
6. The method of claim 1 , wherein in anticipation of deletion of the first chunk at the first zone and in response to a determination that the locally demoted first chunk at the third zone is available at the third zone, the first chunk is decoupled from protection of other chunks including at least the second chunk, the decoupling comprising performing an XOR operation at the third zone on the locally demoted first chunk at the third zone and the XOR chunk to create either a partial XOR chunk or the second chunk.
7. The method of claim 1 , wherein in anticipation of deletion of the first chunk at the first zone and in response to a determination that the locally demoted second chunk at the third zone is available at the third zone, the first chunk is decoupled from protection of other chunks including at least the second chunk, the decoupling comprising either marking the locally demoted second data chunk at the third zone as a regular chunk or creating a partial XOR chunk based at least in part on the locally demoted second chunk at the third zone, the partial XOR chunk being decoupled from the first chunk.
8. The method of claim 1 , wherein in response to a client read request at the third zone comprising a read operation of the first chunk and to a determination that the locally demoted first chunk at the third zone is available at the third zone, the locally demoted first chunk at the third zone is used as a read cache in fulfillment of the client read request.
9. A non-transitory machine-readable medium having instructions stored therein which, when executed by a processor, cause the processor to perform storage operations, the operations comprising:
replicating a first chunk from a first zone of a cloud storage system to a third zone of the cloud storage system;
replicating a second chunk from a second zone of the cloud storage system to the third zone;
creating an exclusive or (XOR) chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk; and
subsequent to the creation of the XOR chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk, marking the replicated first chunk and the replicated second chunk as locally demoted first chunk and locally demoted second chunk, respectively, at the third zone, wherein new data is allowed to be written at the third zone to space corresponding to any locally demoted chunk at the third zone, and wherein data of any locally demoted chunk at the third zone remains available unless and until the corresponding space at the third zone is overwritten.
10. The non-transitory machine-readable medium of claim 9 , wherein the first, second, and third zones are located remotely from each other.
11. The non-transitory machine-readable medium of claim 9 , wherein space occupied by the locally demoted first chunk at the third zone and the locally demoted second chunk at the third zone is categorized as available space when the third zone determines storage space usage.
12. The non-transitory machine-readable medium of claim 9 , wherein in response to a determination that the first chunk is lost at the first zone and the locally demoted first chunk at the third zone is available at the third zone, the first chunk is recovered by marking the locally demoted first chunk at the third zone as a regular chunk or copying the locally demoted first chunk at the third zone to the first zone.
13. The non-transitory machine-readable medium of claim 9 , wherein in response to a determination that the first chunk is lost at the first zone and the locally demoted second chunk at the third zone is available at the third zone, the first chunk is recovered at the third zone based at least in part on the XOR chunk and the locally demoted second chunk at the third zone.
14. The non-transitory machine-readable medium of claim 9 , wherein in anticipation of deletion of the first chunk at the first zone and in response to a determination that the locally demoted first chunk at the third zone is available at the third zone, the first chunk is decoupled from protection of other chunks including at least the second chunk, the decoupling comprising performing an XOR operation at the third zone on the locally demoted first chunk at the third zone and the XOR chunk to create either a partial XOR chunk or the second chunk.
15. The non-transitory machine-readable medium of claim 9 , wherein in anticipation of deletion of the first chunk at the first zone and in response to a determination that the locally demoted second chunk at the third zone is available at the third zone, the first chunk is decoupled from protection of other chunks including at least the second chunk, the decoupling comprising either marking the locally demoted second data chunk at the third zone as a regular chunk or creating a partial XOR chunk based at least in part on the locally demoted second chunk at the third zone, the partial XOR chunk being decoupled from the first chunk.
16. The non-transitory machine-readable medium of claim 9 , wherein in response to a client read request at the third zone comprising a read operation of the first chunk and to a determination that the locally demoted first chunk at the third zone is available at the third zone, the locally demoted first chunk at the third zone is used as a read cache in fulfillment of the client read request.
17. A data processing system, comprising:
a processor; and
a memory coupled to the processor storing instructions which, when executed by the processor, cause the processor to perform storage operations, the operations including:
replicating a first chunk from a first zone of a cloud storage system to a third zone of the cloud storage system;
replicating a second chunk from a second zone of the cloud storage system to the third zone;
creating an exclusive or (XOR) chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk; and
subsequent to the creation of the XOR chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk, marking the replicated first chunk and the replicated second chunk as locally demoted first chunk and locally demoted second chunk, respectively, at the third zone, wherein new data is allowed to be written at the third zone to space corresponding to any locally demoted chunk at the third zone, and wherein data of any locally demoted chunk at the third zone remains available unless and until the corresponding space at the third zone is overwritten.
18. The data processing system of claim 17 , wherein the first, second, and third zones are located remotely from each other.
19. The data processing system of claim 17 , wherein space occupied by the locally demoted first chunk and the locally demoted second chunk is categorized as available space at the third zone when the third zone determines storage space usage.
20. The data processing system of claim 17 , wherein in response to a determination that the first chunk is lost at the first zone and the locally demoted first chunk at the third zone is available at the third zone, the first chunk is recovered by marking the locally demoted first chunk at the third zone as a regular chunk or copying the locally demoted first chunk at the third zone to the first zone.
21. The data processing system of claim 17 , wherein in response to a determination that the first chunk is lost at the first zone and the locally demoted second chunk at the third zone is available at the third zone, the first chunk is recovered at the third zone based at least in part on the XOR chunk and the locally demoted second chunk at the third zone.
22. The data processing system of claim 17 , wherein in anticipation of deletion of the first chunk at the first zone and in response to a determination that the locally demoted first chunk at the third zone is available at the third zone, the first chunk is decoupled from protection of other chunks including at least the second chunk, the decoupling comprising performing an XOR operation at the third zone on the locally demoted first chunk at the third zone and the XOR chunk to create either a partial XOR chunk or the second chunk.
23. The data processing system of claim 17 , wherein in anticipation of deletion of the first chunk at the first zone and in response to a determination that the locally demoted second chunk at the third zone is available at the third zone, the first chunk is decoupled from protection of other chunks including at least the second chunk, the decoupling comprising either marking the locally demoted second data chunk at the third zone as a regular chunk or creating a partial XOR chunk based at least in part on the demoted second chunk, the partial XOR chunk being decoupled from the first chunk.
24. The data processing system of claim 17 , wherein in response to a client read request at the third zone comprising a read operation of the first chunk and to a determination that the locally demoted first chunk at the third zone is used is available at the third zone, the locally demoted first chunk at the third zone as a read cache in fulfillment of the client read request.