Method and system for redundant distribution and reconstruction of storage metadata
In general, the invention relates to a method for managing data, the method includes obtaining data from a host, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk, storing the storage metadata in an accelerator pool, storing, across a plurality of fault domains, the plurality of deduplicated data chunks and the at least one parity chunk, and initiating storage metadata distribution on the storage metadata across the plurality of fault domains.
1. A method for managing data, the method comprising:
obtaining the data from a host;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk;
storing the storage metadata in an accelerator pool, wherein the accelerator pool comprises a plurality of high performance nodes;
storing, across a plurality of fault domains, the plurality of deduplicated data chunks and the at least one parity chunks,
wherein a non-accelerator pool comprises a plurality of data nodes that provide lower storage performance than the plurality of high performance nodes,
wherein the plurality of fault domains is implemented using the plurality of data nodes;
initiating storage metadata distribution on the storage metadata across the plurality of fault domains;
after initiating the storage metadata distribution, identifying a storage metadata failure of the storage metadata in the accelerator pool;
in response to the storage metadata failure, sending a first storage metadata request to a first fault domain of the plurality of fault domains for a first portion of a copy of the storage metadata;
sending a second storage metadata request to a second fault domain of the plurality of fault domains for a second portion of the copy of the storage metadata;
obtaining, in response to the first storage metadata request, the first portion from the first fault domain,
obtaining, in response to the second metadata storage request, the second portion from the second fault domain; and
performing a storage metadata reconstruction of storage metadata on the accelerator pool using the first portion and the second portion.
2. The method of claim 1 , wherein each deduplicated data chunk of the plurality of data chunks is stored in a unique fault domain of the plurality of fault domains, and wherein a copy of the storage metadata is stored in each fault domain of the plurality of fault domains.
3. The method of claim 1 ,
wherein storing the plurality of deduplicated data chunks and the at least one parity chunk comprises: storing a deduplicated data chunk of the plurality of deduplicated data chunks on a first data node in a fault domain of the plurality of fault domains,
wherein initiating storage metadata distribution on the storage metadata across the plurality of fault domains comprises: initiating storage of a copy of the storage metadata on a second data node in the fault domain.
4. The method of claim 1 , wherein the storage metadata includes at least location information of: at least one of the plurality of deduplicated data chunks and of the at least one parity chunk.
5. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing data, the method comprising:
obtaining the data from a host;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk;
storing the storage metadata in an accelerator pool, wherein the accelerator pool comprises a plurality of high performance nodes;
storing, across a plurality of fault domains, the plurality of deduplicated data chunks and the at least one parity chunk,
wherein a non-accelerator pool comprises a plurality of data nodes that provide lower storage performance than the plurality of high performance nodes,
wherein the plurality of fault domains is implemented using the plurality of data nodes;
initiating storage metadata distribution on the storage metadata across the plurality of fault domains;
after initiating the storage metadata distribution, identifying a storage metadata failure of the storage metadata in the accelerator pool;
in response to the storage metadata failure, sending a first storage metadata request to a first fault domain of the plurality of fault domains for a first portion of a copy of the storage metadata;
sending a second storage metadata request to a second fault domain of the plurality of fault domains for a second portion of the copy of the storage metadata;
obtaining, in response to the first storage metadata request, the first portion from the first fault domain,
obtaining, in response to the second storage metadata request, the second portion from the second fault domain; and
performing a storage metadata reconstruction of storage metadata on the accelerator pool using the first portion and the second portion.
6. The non-transitory computer readable medium of claim 5 , wherein each deduplicated data chunk of the plurality of data chunks is stored in a unique fault domain of the plurality of fault domains, and wherein a copy of the storage metadata is stored in a second data node of each fault domain of the plurality of fault domains.
7. The non-transitory computer readable medium of claim 5 ,
wherein storing the plurality of deduplicated data chunks and the at least one parity chunk comprises: storing a deduplicated data chunk of the plurality of deduplicated data chunks on a first data node in a fault domain of the plurality of fault domains,
wherein initiating storage metadata distribution on the storage metadata across the plurality of fault domains comprises: initiating storage of a copy of the storage metadata on a second data node in the fault domain.
8. The non-transitory computer readable medium of claim 5 , wherein the storage metadata includes at least location information of: at least one of the plurality of deduplicated data chunks and of the at least one parity chunk.
9. A data cluster, comprising:
a host;
a non-accelerator pool comprising a plurality of low performance nodes wherein a plurality of fault domains is implemented using the plurality of low performance nodes; and
an accelerator pool comprising a plurality of data nodes, wherein the plurality of data nodes provides higher storage performance than the plurality of low performance nodes,
wherein a data node of the plurality of data nodes comprises a processor and memory comprising instructions, which when executed by the processor perform a method, the method comprising:
obtaining data from the host;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generating storage metadata associated with the plurality of deduplicated data chunks and the at least one parity chunk;
storing the storage metadata in the accelerator pool;
storing, across the plurality of fault domains, the plurality of deduplicated data chunks and the at least one parity chunk;
initiating storage metadata distribution on the storage metadata across the plurality of fault domains;
after initiating the storage metadata distribution, identifying a storage metadata failure of the storage metadata in the accelerator pool;
in response to the storage metadata failure, sending a first storage metadata request to a first fault domain of the plurality of fault domains for a first portion of a copy of the storage metadata;
sending a second storage metadata request to a second fault domain of the plurality of fault domains for a second portion of the copy of the storage metadata;
obtaining, in response to the first storage metadata request, the first portion from the first fault domain,
obtaining, in response to the second storage metadata request, the second portion from the second fault domain; and
performing a storage metadata reconstruction of storage metadata on the accelerator pool using the first portion and the second portion.
10. The data cluster of claim 9 ,
wherein storing the plurality of deduplicated data chunks and the at least one parity chunk comprises: storing a deduplicated data chunk of the plurality of deduplicated data chunks on a first data node in a fault domain of the plurality of fault domains,
wherein initiating storage metadata distribution on the storage metadata across the plurality of fault domains comprises: initiating storage of a copy of the storage metadata on a second data node in the fault domain.