Method and system for accessing data stored in data cache with fault tolerance
According to some embodiments, a backup storage system receives a request from a client to access a data segment. The system looks up an in-memory index for a first entry based on a fingerprint of the data segment, such that the in-memory index includes a number of entries, each entry mapping a fingerprint to a storage location of a solid state device (SSD) storing a corresponding data segment. In the event a first entry is found, the system retrieves the data segment from the SSD device based on the first entry. In the event of a read failure while accessing the SSD device, the system determines failed data blocks of the SSD device associated with the failure event and invalidates the failed data blocks of the SSD device such that subsequent access to the failed data blocks are denied without having to access a corresponding SSD device.
1. A computer-implemented method, the method comprising:
receiving a request from a client to access a data segment associated with a file object of a storage system;
looking up an in-memory index for a first entry based on a fingerprint of the data segment, wherein the in-memory index includes a plurality of entries, each entry mapping the fingerprint of the data segment being contained in a log-structured container to a storage location of a solid state device (SSD) storing a corresponding data segment;
in the event a first entry is found, retrieving the data segment from the SSD based on the first entry;
in the event of a read failure while accessing the SSD,
determining one or more failed data blocks of the SSD associated with the failure event; and
invalidating the failed data blocks of the SSD in response to determining the one or more failed data blocks, the invalidating of the failed data blocks of the SSD being recorded such that subsequent access to the failed data blocks are denied without having to accessing a corresponding SSD, wherein invalidating the failed data blocks comprises:
recording on a descriptive data block of the SSD, information of the failed data blocks of the SSD, such that subsequent read requests will check the descriptive data block for validity of requested data blocks, wherein the information of the failed data blocks include a table of entries recording a SSD device identifier (ID), a write-evict unit (WEU) ID identifying a WEU storing the requested data segment, and a block range or block size that corresponds to invalid blocks; and
propagating the recorded descriptive data block associated with the SSD to each disk volume of a plurality of disk volumes of the SSD, wherein each disk volume of the plurality of disk volumes of the SSD reserves the descriptive data block, such that each of the reserved descriptive data block includes a mirror image of the recorded descriptive data block having information of the failed data blocks of the SSD, each of the reserved descriptive data block marking an invalid disk volume of the plurality of disk volumes of the SSD.
2. The computer-implemented method of claim 1 , further comprising: propagating a copy of the descriptive data block in-memory such that validity of requested data blocks can be determined in-memory without accessing the SSD.
3. The computer-implemented method of claim 1 , further comprising:
in the event of a read failure, retrieving the requested data segment from a storage unit of the storage system; and
upon receiving the requested data segment form the storage unit, populating the SSD with the requested data segment.
4. The computer-implemented method of claim 3 , further comprising:
detecting a failure event while writing a new data segment to the SSD; and
in the event of a write failure, invalidating the data blocks of the SSD associated with the write request such that subsequent access to the failed data blocks are unavailable.
5. The computer-implemented method of claim 4 , further comprising:
writing the new data segment to a second SSD; and updating the in-memory index to map the fingerprint of the new data segment to the second SSD, the fingerprint of the new data segment being contained in a log-structured container, the log-structured container being structured into a metadata section and a compression region section.
6. The computer-implemented method of claim 1 , wherein a failure of the cache memory device comprises: a checksum failure, a volume failure, a corruption, and a disk failure.
7. The computer-implemented method of claim 1 , further comprising:
upon detection of an event associated with a hot-plug of one of a plurality of individual SSDs, updating the descriptive data block of the SSD to remove information of failed data blocks associated with one or more disk volumes of the hot-plugged individual SSD.
8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations to sort data elements in a data array, the operations comprising:
receiving a request from a client to access a data segment associated with a file object of a storage system;
looking up an in-memory index for a first entry based on a fingerprint of the data segment, wherein the in-memory index includes a plurality of entries, each entry mapping the fingerprint of the data segment being contained in a log-structured container to a storage location of a solid state device (SSD) storing a corresponding data segment;
in the event a first entry is found, retrieving the data segment from the SSD based on the first entry;
in the event of a read failure while accessing the SSD,
determining one or more failed data blocks of the SSD associated with the failure event; and
invalidating the failed data blocks of the SSD in response to determining the one or more failed data blocks, the invalidating of the failed data blocks of the SSD being recorded such that subsequent access to the failed data blocks are denied without having to accessing a corresponding SSD, wherein invalidating the failed data blocks comprises:
recording on a descriptive data block of the SSD, information of the failed data blocks of the SSD, such that subsequent read requests will check the descriptive data block for validity of requested data blocks, wherein the information of the failed data blocks include a table of entries recording a SSD device identifier (ID), a write-evict unit (WEU) ID identifying a WEU storing the requested data segment, and a block range or block size that corresponds to invalid blocks; and
propagating the recorded descriptive data block associated with the SSD to each disk volume of a plurality of disk volumes of the SSD, wherein each disk volume of the plurality of disk volumes of the SSD reserves the descriptive data block, such that each of the reserved descriptive data block includes a mirror image of the recorded descriptive data block having information of the failed data blocks of the SSD, each of the reserved descriptive data block marking an invalid disk volume of the plurality of disk volumes of the SSD.
9. The non-transitory machine-readable medium of claim 8 , further comprising: propagating a copy of the descriptive data block in-memory such that validity of requested data blocks can be determined in-memory without accessing the SSD.
10. The non-transitory machine-readable medium of claim 8 , further comprising:
in the event of a read failure, retrieving the requested data segment from a storage unit of the storage system; and
upon receiving the requested data segment form the storage unit, populating the SSD with the requested data segment.
11. The non-transitory machine-readable medium of claim 10 , further comprising:
detecting a failure event while writing a new data segment to the SSD; and
in the event of a write failure, invalidating the data blocks of the SSD associated with the write request such that subsequent access to the failed data blocks are unavailable.
12. The non-transitory machine-readable medium of claim 11 , further comprising:
writing the new data segment to a second SSD; and updating the in-memory index to map the fingerprint of the new data segment to the second SSD, the fingerprint of the new data segment being contained in a log-structured container, the log-structured container being structured into a metadata section and a compression region section.
13. The non-transitory machine-readable medium of claim 8 , wherein a failure of the cache memory device comprises: a checksum failure, a volume failure, a corruption, and a disk failure.
14. The non-transitory machine-readable medium of claim 8 , further comprising:
upon detection of an event associated with a hot-plug of one of a plurality of individual SSDs, updating the descriptive data block of the SSD to remove information of failed data blocks associated with one or more disk volumes of the hot-plugged individual SSD.
15. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including receiving a request from a client to access a data segment associated with a file object of a storage system;
looking up an in-memory index for a first entry based on a fingerprint of the data segment, wherein the in-memory index includes a plurality of entries, each entry mapping the fingerprint of the data segment being contained in a log-structured container to a storage location of a solid state device (SSD) storing a corresponding data segment;
in the event a first entry is found, retrieving the data segment from the SSD based on the first entry;
in the event of a read failure while accessing the SSD,
determining one or more failed data blocks of the SSD associated with the failure event; and
invalidating the failed data blocks of the SSD in response to determining the one or more failed data blocks, the invalidating of the failed data blocks of the SSD being recorded such that subsequent access to the failed data blocks are denied without having to accessing a corresponding SSD, wherein invalidating the failed data blocks comprises:
recording on a descriptive data block of the SSD, information of the failed data blocks of the SSD, such that subsequent read requests will check the descriptive data block for validity of requested data blocks, wherein the information of the failed data blocks include a table of entries recording a SSD device identifier (ID), a write-evict unit (WEU) ID identifying a WEU storing the requested data segment, and a block range or block size that corresponds to invalid blocks; and
propagating the recorded descriptive data block associated with the SSD to each disk volume of a plurality of disk volumes of the SSD, wherein each disk volume of the plurality of disk volumes of the SSD reserves the descriptive data block, such that each of the reserved descriptive data block includes a mirror image of the recorded descriptive data block having information of the failed data blocks of the SSD, each of the reserved descriptive data block marking an invalid disk volume of the plurality of disk volumes of the SSD.
16. The data processing system of claim 15 , further comprising: propagating a copy of the descriptive data block in-memory such that validity of requested data blocks can be determined in-memory without accessing the SSD.
17. The data processing system of claim 15 , further comprising:
in the event of a read failure, retrieving the requested data segment from a storage unit of the storage system; and
upon receiving the requested data segment form the storage unit, populating the SSD with the requested data segment.
18. The data processing system of claim 17 , further comprising:
detecting a failure event while writing a new data segment to the SSD; and
in the event of a write failure, invalidating the data blocks of the SSD associated with the write request such that subsequent access to the failed data blocks are unavailable.