Method and system for managing a spare fault domain in a multi-fault domain data cluster
In general, the invention relates to a method for managing data. The method includes detecting a fault domain failure in a fault domain of a plurality of fault domains, and in response to the detecting, initiating a rebuilding of data in a spare fault domain of using checkpoint metadata and slice metadata, wherein the data is a copy of data that was stored in the fault domain and wherein the spare fault domain is not one of the plurality of fault domains.
1. A method for managing data, the method comprising:
detecting a fault domain failure in a fault domain of a plurality of fault domains; and
in response to the detecting, initiating a rebuilding of data in a spare fault domain using checkpoint metadata and slice metadata, wherein the data is a copy of data that was stored in the fault domain and wherein the spare fault domain is not one of the plurality of fault domains,
wherein prior to detecting the fault domain failure:
identifying a checkpoint zone, wherein the checkpoint zone comprises chunks stored across the plurality of fault domains;
selecting a subset of the chunks, wherein the checkpoint metadata is based on the selected subset of the chunks; and
initiating writing of copies of the subset of the chunks to the spare fault domain.
2. The method of claim 1 , wherein the subset of the chunks comprises at least one data chunk and at least one parity chunk.
3. The method of claim 1 , wherein the subset of the chunks comprises at least one chunk from each of the plurality of fault domains.
4. The method of claim 1 ,
wherein the checkpoint zone comprises a plurality of slices,
wherein each of the chunks is associated with one of the plurality of slices; and
wherein a cardinality of the plurality of slices is based on a checkpoint threshold.
5. The method of claim 4 , wherein at least one chunk associated with each of the plurality of slices is a parity chunk.
6. The method of claim 5 , wherein the slice metadata specifies which of the chunks are associated with each of the plurality of slices.
7. A system, comprising:
a processor;
a data processor, which when executed by the processor performs a method, the method comprising:
detecting a fault domain failure in a fault domain of a plurality of fault domains; and
in response to the detecting, initiating a rebuilding of data in a spare fault domain using checkpoint metadata and slice metadata, wherein the data is a copy of data that was stored in the fault domain and wherein the spare fault domain is not one of the plurality of fault domains,
wherein prior to detecting the fault domain failure:
identifying a checkpoint zone, wherein the checkpoint zone comprises chunks stored across the plurality of fault domains;
selecting a subset of the chunks, wherein the checkpoint metadata is based on the selected subset of the chunks; and
initiating writing of copies of the subset of the chunks to the spare fault domain.
8. The system of claim 7 , wherein the subset of the chunks comprise at least one data chunk and at least one parity chunk.
9. The system of claim 7 , wherein the subset of the chunks comprises at least one chunk from each of the plurality of fault domains.
10. The system of claim 7 ,
wherein the checkpoint zone comprises a plurality of slices,
wherein each of the chunks is associated with one of the plurality of slices; and
wherein a cardinality of the plurality of slices is based on a checkpoint threshold.
11. The system of claim 10 , wherein at least one chunk associated with each of the plurality of slices is a parity chunk.
12. The system of claim 11 , wherein the slice metadata specifies which of the chunks are associated with each of the plurality of slices.
13. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method, the method comprising:
detecting a fault domain failure in a fault domain of a plurality of fault domains; and
in response to the detecting, initiating a rebuilding of data in a spare fault domain using checkpoint metadata and slice metadata, wherein the data is a copy of data that was stored in the fault domain and wherein the spare fault domain is not one of the plurality of fault domains,
wherein prior to detecting the fault domain failure:
identifying a checkpoint zone, wherein the checkpoint zone comprises chunks stored across the plurality of fault domains;
selecting a subset of the chunks, wherein the checkpoint metadata is based on the selected subset of the chunks; and
initiating writing of copies of the subset of the chunks to the spare fault domain.
14. The non-transitory computer readable medium of claim 11 , wherein the subset of the chunks comprise at least one data chunk and at least one parity chunk.
15. The non-transitory computer readable medium of claim 13 , wherein the subset of the chunks comprises at least one chunk from each of the plurality of fault domains.
16. The non-transitory computer readable medium of claim 13 ,
wherein the checkpoint zone comprises a plurality of slices,
wherein each of the chunks is associated with one of the plurality of slices; and
wherein a cardinality of the plurality of slices is based on a checkpoint threshold.
17. The non-transitory computer readable medium of claim 16 , wherein at least one chunk associated with each of the plurality of slices is a parity chunk.