Method and system for offloading a continuous health-check and reconstruction of data in a non-accelerator pool
A method for managing data includes identifying, by a compute acceleration device (CAD), a first chunk to be health-checked using storage metadata, generating a first chunk identifier using the first chunk, making a determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata, and in response to the determination: obtaining a plurality of chunks associated with the first chunk, regenerating the first chunk using the plurality of chunks to generate a new first chunk, storing the new first chunk in a data node, wherein the CAD is executing in the data node, updating the storage metadata based on storage of the new first chunk to obtain updated storage metadata, and sending a copy of the updated storage metadata to at least one other CAD in a second data node.
1. A data cluster, comprising:
a non-accelerator pool comprising a plurality of data nodes;
wherein a data node of the plurality of data nodes comprises a processor, a compute acceleration device (CAD), wherein the CAD is located in a first fault domain, and
wherein the CAD is programmed to:
identify a first chunk to be health-checked using storage metadata;
generate a first chunk identifier using the first chunk;
make a determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata; and
in response to the determination:
obtain a plurality of chunks associated with the first chunk;
regenerate the first chunk using the plurality of chunks to generate a new first chunk;
store the new first chunk in the data node;
update the storage metadata based on storage of the new first chunk to obtain updated storage metadata; and
send a copy of the updated storage metadata to at least a second CAD in a second data node of the plurality of data nodes, wherein the second CAD is located in a second fault domain.
2. The data cluster of claim 1 , wherein the first chunk identifier is a fingerprint of the first chunk.
3. The data cluster of claim 1 , wherein updating the storage metadata comprises:
updating the storage metadata to include a third chunk identifier of the new first chunk; and
updating the storage metadata to include a storage location of the new first chunk.
4. The data cluster of claim 1 , wherein the first chunk is stored on a persistent storage device in the data node and wherein the CAD is operatively connected to the persistent storage device.
5. The data cluster of claim 1 , wherein the storage metadata is stored in the CAD.
6. The data cluster of claim 1 , wherein the first chunk is stored on a first persistent storage device in the data node and the wherein the new first chunk is stored on a second persistent storage device in the data node.
7. A method for managing data, the method comprising:
identifying, by a compute acceleration device (CAD), a first chunk to be health-checked using storage metadata, wherein the CAD is located on a first fault domain;
generating a first chunk identifier using the first chunk;
making a determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata; and
in response to the determination:
obtaining a plurality of chunks associated with the first chunk;
regenerating the first chunk using the plurality of chunks to generate a new first chunk;
storing the new first chunk in a data node, wherein the CAD is executing in the data node;
updating the storage metadata based on storage of the new first chunk to obtain updated storage metadata; and
sending a copy of the updated storage metadata to at least a second CAD in a second data node, wherein the second CAD is located in a second fault domain.
8. The method of claim 7 , wherein the first chunk identifier is a fingerprint of the first chunk.
9. The method of claim 7 , wherein updating the storage metadata comprises:
updating the storage metadata to include a third chunk identifier of the new first chunk; and
updating the storage metadata to include a storage location of the new first chunk.
10. The method of claim 7 , wherein the first chunk is stored on a persistent storage device in the data node and wherein the CAD is operatively connected to the persistent storage device.
11. The method of claim 7 , wherein the storage metadata is stored in the CAD.
12. The method of claim 7 , wherein the first chunk is stored on a first persistent storage device in the data node and the wherein the new first chunk is stored on a second persistent storage device in the data node.
13. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing data, the method comprising:
identifying, by a compute acceleration device (CAD), a first chunk to be health-checked using storage metadata, wherein the CAD is located in a first fault domain;
generating a first chunk identifier using the first chunk;
making a determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata; and
in response to the determination:
obtaining a plurality of chunks associated with the first chunk;
regenerating the first chunk using the plurality of chunks to generate a new first chunk;
storing the new first chunk in a data node, wherein the CAD is executing in the data node;
updating the storage metadata based on storage of the new first chunk to obtain updated storage metadata; and
sending a copy of the updated storage metadata to at least a second CAD in a second data node, wherein the second CAD is located in a second fault domain.
14. The non-transitory computer readable medium of claim 13 , wherein the first chunk identifier is a fingerprint of the first chunk.
15. The non-transitory computer readable medium of claim 13 , wherein updating the storage metadata comprises:
updating the storage metadata to include a third chunk identifier of the new first chunk; and
updating the storage metadata to include a storage location of the new first chunk.
16. The non-transitory computer readable medium of claim 13 , wherein the first chunk is stored on a persistent storage device in the data node and wherein the CAD is operatively connected to the persistent storage device.
17. The non-transitory computer readable medium of claim 13 , wherein the first chunk is stored on a first persistent storage device in the data node and the wherein the new first chunk is stored on a second persistent storage device in the data node.