Method and system for inline deduplication using erasure coding
A method includes obtaining a data, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, and storing, across a plurality of nodes, the plurality of deduplicated data chunks and the at least one parity chunk.
1. A method for storing data, the method comprising:
obtaining data;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, wherein deduplicating the plurality of data chunks to obtain the plurality of deduplicated data chunks is performed after a parity value for the plurality of data chunks is generated by performing an erasure coding procedure; and
storing, across a plurality of nodes, the plurality of deduplicated data chunks and the at least one parity chunk.
2. The method of claim 1 , wherein the erasure coding procedure is applied by a deduplicator executing on a node in an accelerator pool, wherein the plurality of nodes is located is a non-accelerator pool, and wherein a data cluster comprises the accelerator pool and the non-accelerator pool.
3. The method of claim 1 , wherein applying the erasure coding procedure comprises:
dividing the data into data chunks;
selecting, from the data chunks, the plurality of data chunks; and
generating the at least one the parity chunk using the plurality of data chunks.
4. The method of claim 1 , wherein the at least one parity chunk comprises a P parity value.
5. The method of claim 1 , wherein the at least one parity chunk comprises a first parity chunk comprising a P parity value and a second parity chunk comprising a Q parity value.
6. The method of claim 1 , wherein each of the plurality of nodes is in a separate fault domain.
7. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for storing, the method comprising:
obtaining data;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, wherein deduplicating the plurality of data chunks to obtain the plurality of deduplicated data chunks is performed after a parity value for the plurality of data chunks is generated by performing an erasure coding procedure; and
storing, across a plurality of nodes, the plurality of deduplicated data chunks and the at least one parity chunk.
8. The non-transitory computer readable medium of claim 7 , wherein the erasure coding procedure is applied by a deduplicator executing on a node in an accelerator pool, wherein the plurality of nodes is located is a non-accelerator pool, and wherein a data cluster comprises the accelerator pool and the non-accelerator pool.
9. The non-transitory computer readable medium of claim 7 , wherein applying the erasure coding procedure comprises:
dividing the data into data chunks;
selecting, from the data chunks, the plurality of data chunks; and
generating the at least one the parity chunk using the plurality of data chunks.
10. The non-transitory computer readable medium of claim 7 , wherein the at least one parity chunk comprises a P parity value.
11. The non-transitory computer readable medium of claim 7 , wherein the at least one parity chunk comprises a first parity chunk comprising a P parity value and a second parity chunk comprising a Q parity value.
12. The non-transitory computer readable medium of claim 7 , wherein each of the plurality of nodes is in a separate fault domain.
13. A data cluster, comprising:
a plurality of data nodes comprising an accelerator pool and a non-accelerator pool, wherein the accelerator pool comprises a data node, and the non-accelerator pool comprises a plurality of data nodes;
wherein the data node of the plurality of data nodes is programmed to:
obtain data;
apply an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicate the plurality of data chunks to obtain a plurality of deduplicated data chunks, wherein deduplicating the plurality of data chunks to obtain the plurality of deduplicated data chunks is performed after a parity value for the plurality of data chunks is generated by performing an erasure coding procedure; and
store, across the plurality of nodes, the plurality of deduplicated data chunks and the at least one parity chunk.
14. The data cluster of claim 13 , wherein applying the erasure coding procedure comprises:
dividing the data into data chunks;
selecting, from the data chunks, the plurality of data chunks; and
generating the at least one the parity chunk using the plurality of data chunks.
15. The data cluster of claim 13 , wherein the at least one parity chunk comprises a P parity value.
16. The data cluster of claim 13 , wherein the at least one parity chunk comprises a first parity chunk comprising a P parity value and a second parity chunk comprising a Q parity value.
17. The data cluster of claim 13 , wherein each of the plurality of nodes is in a separate fault domain.