Method and system for offloading a continuous health-check and reconstruction of data in a data cluster
A method for managing data includes obtaining, by a storage controller, data from a host, applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk, deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks, generating storage metadata associated with the plurality of data chunks and the at least one parity chunk, storing, across a plurality of persistent storage devices, the plurality of deduplicated data chunks and the at least one parity chunk, wherein the plurality of persistent storage devices is operatively connected to the storage controller and a second storage controller, storing the storage metadata in the storage controller, and sending a copy of the storage metadata to the second storage controller.
1. A method for managing data, the method comprising:
obtaining, by a storage controller, data from a host;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generating storage metadata associated with the plurality of data chunks and the at least one parity chunk;
storing, across a plurality of persistent storage devices, the plurality of deduplicated data chunks and the at least one parity chunk, wherein the plurality of persistent storage devices is operatively connected to the storage controller and a second storage controller;
storing the storage metadata in the storage controller; and
sending a copy of the storage metadata to the second storage controller.
2. The method of claim 1 , further comprising:
identifying, by the storage controller, a state of the second storage controller;
making a first determination that the second controller is not in a functioning state;
initiating, in response to the first determination, a health-check on at least one chunk in a first persistent storage device assigned to the second storage controller, wherein the first persistent storage device is one of the plurality of persistent storage devices.
3. The method of claim 2 , wherein initiating the health-check on the at least one chunk comprises:
obtaining a first chunk from the first persistent storage device;
generating a first chunk identifier using the first chunk;
making a second determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata; and
in response to the second determination:
obtaining a plurality of chunks associated with the first chunk;
regenerating the first chunk using the plurality of chunks to generate a new first chunk;
storing the new first chunk in a second persistent storage device of the plurality of persistent storage devices;
updating the storage metadata based on storage of the new first chunk.
4. The method of claim 3 , wherein the second persistent storage device is assigned to the first storage controller.
5. The method of claim 3 , wherein updating the storage metadata comprises:
updating the second chunk identifier with a third chunk identifier of the new first chunk; and
updating a storage location of the first chunk with a storage location of the new first chunk.
6. The method of claim 1 , wherein the storage metadata is stored in memory of the first storage controller.
7. The method of claim 1 , wherein a first subset of the plurality of persistent storage devices is assigned to the first storage controller for health-checking and wherein a second subset of the plurality of persistent storage devices is assigned to the second storage controller for health-checking.
8. The method of claim 7 , wherein a first portion of the deduplicated data chunks are stored in the first subset and a second portion of the deduplicated data chunks are stored in the second subset.
9. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing data, the method comprising:
obtaining, by a storage controller, data from a host;
applying an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicating the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generating storage metadata associated with the plurality of data chunks and the at least one parity chunk;
storing, across a plurality of persistent storage devices, the plurality of deduplicated data chunks and the at least one parity chunk, wherein the plurality of persistent storage devices is operatively connected to the storage controller and a second storage controller;
storing the storage metadata in the storage controller; and
sending a copy of the storage metadata to the second storage controller.
10. The non-transitory computer readable medium of claim 9 , the method further comprising:
identifying, by the storage controller, a state of the second storage controller;
making a first determination that the second controller is not in a functioning state;
initiating, in response to the first determination, a health-check on at least one chunk in a first persistent storage device assigned to the second storage controller, wherein the first persistent storage device is one of the plurality of persistent storage devices.
11. The non-transitory computer readable medium of claim 10 , wherein initiating the health-check on the at least one chunk comprises:
obtaining a first chunk from the first persistent storage device;
generating a first chunk identifier using the first chunk;
making a second determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata; and
in response to the second determination:
obtaining a plurality of chunks associated with the first chunk;
regenerating the first chunk using the plurality of chunks to generate a new first chunk;
storing the new first chunk in a second persistent storage device of the plurality of persistent storage devices;
updating the storage metadata based on storage of the new first chunk.
12. The non-transitory computer readable medium of claim 11 , wherein the second persistent storage device is assigned to the first storage controller.
13. The non-transitory computer readable medium of claim 11 , wherein updating the storage metadata comprises:
updating the second chunk identifier with a third chunk identifier of the new first chunk; and
updating a storage location of the first chunk with a storage location of the new first chunk.
14. The non-transitory computer readable medium of claim 9 , wherein the storage metadata is stored in memory of the first storage controller.
15. The non-transitory computer readable medium of claim 9 , wherein a first subset of the plurality of persistent storage devices is assigned to the first storage controller for health-checking and wherein a second subset of the plurality of persistent storage devices is assigned to the second storage controller for health-checking.
16. The non-transitory computer readable medium of claim 15 , wherein a first portion of the deduplicated data chunks are stored in the first subset and a second portion of the deduplicated data chunks are stored in the second subset.
17. A data cluster, comprising:
a first storage controller;
a second storage controller; and
a plurality of persistent storage devices,
wherein the first storage controller is programmed to:
obtain data from a host;
apply an erasure coding procedure to the data to obtain a plurality of data chunks and at least one parity chunk;
deduplicate the plurality of data chunks to obtain a plurality of deduplicated data chunks;
generate storage metadata associated with the plurality of data chunks and the at least one parity chunk;
store, across the plurality of persistent storage devices, the plurality of deduplicated data chunks and the at least one parity chunk;
store the storage metadata in the first storage controller; and
send a copy of the storage metadata to the second storage controller.
18. The data cluster of claim 17 , wherein the storage controller is further programmed to:
identify a state of the second storage controller;
make a first determination that the second controller is not in a functioning state;
initiate, in response to the first determination, a health-check on at least one chunk in a first persistent storage device assigned to the second storage controller, wherein the first persistent storage device is one of the plurality of persistent storage devices.
19. The data cluster of claim 18 , wherein initiate the health-check on the at least one chunk comprises:
obtaining a first chunk from the first persistent storage device;
generating a first chunk identifier using the first chunk;
making a second determination that the first chunk identifier does not match a second chunk identifier stored in the storage metadata; and
in response to the second determination:
obtaining a plurality of chunks associated with the first chunk;
regenerating the first chunk using the plurality of chunks to generate a new first chunk;
storing the new first chunk in a second persistent storage device of the plurality of persistent storage devices;
updating the storage metadata based on storage of the new first chunk.
20. The data cluster of claim 17 ,
wherein a first subset of the plurality of persistent storage devices is assigned to the first storage controller for health-checking and wherein a second subset of the plurality of persistent storage devices is assigned to the second storage controller for health-checking, and
wherein a first portion of the deduplicated data chunks are stored in the first subset and a second portion of the deduplicated data chunks are stored in the second subset.