Method and system for container data recovery in a storage system
A computer-implemented method is disclosed. The method starts with determining a first container of a storage system is invalid. The method continues with the storage system setting a data recovery state for the first container to be en-queue, which indicates that data of at least one of the data segments needs to be recovered from the first container, and executing a process to recover any container having an en-queue data recovery state, and for each of the containers, to recover any valid data segment from the corresponding container. The process includes scanning the data segments of the first container to find valid data segments, moving or replicating the valid data segments to a second container, and setting the data recovery state for the first container to be complete once all the valid data segments are moved or replicated to the second container.
1. A computer-implemented method, comprising:
determining that a first container of a deduplication storage system is invalid, wherein the first container contains at least a corrupted or missing data segment, wherein the first container stores data segments of the storage system having a plurality of containers, each container having a plurality of compression regions, each compression region having a plurality of data segments compressed and stored therein, including deduplicated segments, wherein at least one deduplicated segment is referenced by a plurality of files, and wherein determining that the first container of the deduplication storage system being invalid includes:
(i) checking, by a container recovery module during a data scrubbing operation, data integrity of containers of the storage system periodically, and determining the first container being invalid based on the data integrity check of the first container,
(ii) reading, by a deduplication eliminator during a deduplication operation, the first container of the storage system to locate a data segment, and checking a checksum associated with the first container to determine that the first container is invalid, and
(iii) looking up, by an indexing module during an indexing operation, an index of the storage system to locate a specific data segment within the first container based on a fingerprint of the specific data segment, the fingerprint being an identifier based on content of the specific data segment, and identifying that the first container including the specific data segment has been marked as invalid, wherein the data scrubbing operation, the deduplication operation, and the indexing operation are performed independently via different threads;
setting a data recovery state for the first container to be en-queue, which indicates that data of at least one of the data segments needs to be recovered from the first container;
executing a first process to recover any container having an en-queue data recovery state, and for each of the containers having an en-queue data recovery state, to recover any valid data segment from the corresponding container, including:
scanning the data segments of the first container to find valid data segments;
moving or replicating the valid data segments to a second container; and
setting the data recovery state for the first container to be complete once all the valid data segments are moved or replicated to the second container; and
executing a second process to scan any container having a complete data recovery state to reclaim storage space, including:
determining that the data recovery state for the first container is complete;
copying content of the first container to a file;
deleting the first container; and
reclaiming storage space of the first container.
2. The computer-implemented method of claim 1 , wherein the second process further comprises:
checking whether fingerprints associated with the first container are associated with any valid container prior to deleting the first container; and
reporting a missing data segment corresponding to any data segment with fingerprint not associated with any valid container.
3. The computer-implemented method of claim 1 , wherein the second process further comprises:
sending a fingerprint that is not associated with any valid container to a replication storage, wherein the replication storage searches for a data segment corresponding to the fingerprint within and sends the data segment corresponding to the fingerprint to the storage system if the data segment is in a third container that is valid; and
writing the received data segment received from the replication storage to the storage system.
4. A storage system, comprising:
a memory configured to store data and instructions; and
a processor configured to execute a deduplication storage engine stored in the memory, the deduplication storage engine configured to:
determine that a first container of a deduplication storage system is invalid, wherein the first container contains at least a corrupted or missing data segment, wherein the first container stores data segments of the storage system having a plurality of containers, each container having a plurality of compression regions, each compression region having a plurality of data segments compressed and stored therein, including deduplicated segments, wherein at least one deduplicated segment is referenced by a plurality of files, and wherein determining that the first container of the deduplication storage system being invalid includes:
(i) checking, by a container recovery module during a data scrubbing operation, data integrity of containers of the storage system periodically, and determining the first container being invalid based on the data integrity check of the first container,
(ii) reading, by a deduplication eliminator during a deduplication operation, the first container of the storage system to locate a data segment, and checking a checksum associated with the first container to determine that the first container is invalid, and
(iii) looking up, by an indexing module during an indexing operation, an index of the storage system to locate a specific data segment within the first container based on a fingerprint of the specific data segment, the fingerprint being an identifier based on content of the specific data segment, and identifying that the first container including the specific data segment has been marked as invalid, wherein the data scrubbing operation, the deduplication operation, and the indexing operation are performed independently via different threads; and
set a data recovery state for the first container to be en-queue, which indicates that data of at least one of the data segments needs to be recovered from the first container;
execute a first process to recover any container having an en-queue data recovery state, and for each of the containers having an en-queue data recovery state, to recover any valid data segment from the corresponding container, including:
scan the data segments of the first container to find valid data segments;
move or replicate the valid data segments to a second container; and
set the data recovery state for the first container to be complete once all the valid data segments are moved or replicated to the second container; and
execute a second process to scan any container having a complete data recovery state to reclaim storage space, including:
determining that the data recovery state for the first container is complete;
copying content of the first container to a file;
deleting the first container; and
reclaiming storage space of the first container.
5. The storage system of claim 4 , wherein the second process further comprises:
checking whether fingerprints associated with the first container are associated with any valid container prior to deleting the first container; and
reporting a missing data segment corresponding to any data segment with fingerprint not associated with any valid container.
6. A non-transitory computer-readable medium having instructions stored therein, which when executed by a processor, causes the processor to perform operations, the operations comprising:
determining that a first container of a deduplication storage system is invalid, wherein the first container contains at least a corrupted or missing data segment, wherein the first container stores data segments of the storage system having a plurality of containers, each container having a plurality of compression regions, each compression region having a plurality of data segments compressed and stored therein, including deduplicated segments, wherein at least one deduplicated segment is referenced by a plurality of files, and wherein determining that the first container of the deduplication storage system being invalid includes:
(i) checking, by a container recovery module during a data scrubbing operation, data integrity of containers of the storage system periodically, and determining the first container being invalid based on the data integrity check of the first container,
(ii) reading, by a deduplication eliminator during a deduplication operation, the first container of the storage system to locate a data segment, and checking a checksum associated with the first container to determine that the first container is invalid, and
(iii) looking up, by an indexing module during an indexing operation, an index of the storage system to locate a specific data segment within the first container based on a fingerprint of the specific data segment, the fingerprint being an identifier based on content of the specific data segment, and identifying that the first container including the specific data segment has been marked as invalid, wherein the data scrubbing operation, the deduplication operation, and the indexing operation are performed independently via different threads;
setting a data recovery state for the first container to be en-queue, which indicates that data of at least one of the data segments needs to be recovered from the first container;
executing a first process to recover any container having an en-queue data recovery state, and for each of the containers having an en-queue data recovery state, to recover any valid data segment from the corresponding container, including:
scanning the data segments of the first container to find valid data segments;
moving or replicating the valid data segments to a second container; and
setting the data recovery state for the first container to be complete once all the valid data segments are moved or replicated to the second container; and
executing a second process to scan any container having a complete data recovery state to reclaim storage space, including:
determining that the data recovery state for the first container is complete;
copying content of the first container to a file;
deleting the first container; and
reclaiming storage space of the first container.
7. The non-transitory computer-readable medium of claim 6 , wherein the second process further comprises:
checking whether fingerprints associated with the first container are associated with any valid container prior to deleting the first container; and
reporting a missing data segment corresponding to any data segment with fingerprint not associated with any valid container.
8. The storage system of claim 4 , wherein the second process further comprises:
sending a fingerprint that is not associated with any valid container to a replication storage, wherein the replication storage searches for a data segment corresponding to the fingerprint within and sends the data segment corresponding to the fingerprint to the storage system if the data segment is in a third container that is valid; and
writing the received data segment received from the replication storage to the storage system.
9. The non-transitory computer-readable medium of claim 6 , wherein the second process further comprises:
sending a fingerprint that is not associated with any valid container to a replication storage, wherein the replication storage searches for a data segment corresponding to the fingerprint within and sends the data segment corresponding to the fingerprint to the storage system if the data segment is in a third container that is valid; and
writing the received data segment received from the replication storage to the storage system.