Method to serve restores from remote high-latency tiers by reading available data from a local low-latency tier in a deduplication appliance
Embodiments are described for performing file restores from remote high-latency storage tiers by reading available data from a local low-latency tier in a deduplication appliance. A request to restore a previously segmented and deduplicated file can be received by a storage appliance from an application, each segment having a fingerprint. The name of the file can be looked up in an index on the storage appliance, and a first batch of fingerprints of segments of the file can be retrieved from the index. Each fingerprint can be looked up in metadata in the index to determine whether the segment corresponding to the fingerprint is available locally and therefore need not be retrieved locally. A list of local and remote prefetch segments is generated, and a prefetch request is generated for each list, if non-empty. Use of the prefetch scheme can be dynamically turned on or off.
1. A computer-implemented comprising:
receiving, by a storage appliance, a request to restore a file to the storage appliance, the file having been previously segmented into a plurality of deduplicated segments, each deduplicated segment having a fingerprint;
determining, by the storage appliance, a tier of storage where all deduplicated segments of the requested file are stored;
for each batch of fingerprints of a plurality of batches of fingerprints, iteratively performing:
determining a hit-rate as a first ratio of segments associated with a previous batch of fingerprints that are stored on the storage appliance vs. stored on the tier storage;
retrieving, from the storage appliance, the batch of fingerprints, each fingerprint in the batch of fingerprints identifying a deduplicated segment of the file;
for each fingerprint in the batch of fingerprints, accessing an index of the storage appliance to determine where the segment associated with the fingerprint is stored;
determining whether an advanced direct restore (ADR) was enabled or disabled for the previous batch of fingerprints;
in response to determining that the ADR was disabled for the previous batch of fingerprints, determining the hit-rate as a second ratio of segments associated with a portion of fingerprints of the batch of fingerprints that are stored on the storage appliance vs. stored on the tier storage;
determining whether the hit-rate is below a threshold;
in response to determining that the hit-rate is above the threshold, enabling the ADR for the batch of fingerprints and prefetching segments associated with the batch of fingerprints by retrieving at least one segment associated with a fingerprint in the batch of fingerprints from the storage appliance and retrieving at least one other segment associated with one other fingerprint in the batch of fingerprints from the tier of storage where all segments of the file are stored; and
in response to determining that the hit-rate is below the threshold, disabling the ADR and retrieving all of the segments associated with the batch of fingerprints from the tier of storage.
2. The method of claim 1 , wherein the tier of storage where all deduplicated segments of the file are located is distinct from the storage appliance.
3. The method of claim 1 , wherein all segments of the file are located on a remote tier of storage that comprises a cloud storage service.
4. The method of claim 1 , wherein the request is received from an application in the context of a storage appliance input/output (I/O) operation from the application.
5. The method of claim 1 , wherein the prefetched segments are read and provided to an application that requested that the file be restored.
6. The method of claim 1 , wherein the batch of fingerprints are ordered in a file order of the prefetched segments.
7. A non-transitory computer-readable medium programmed with executable instructions that, when executed by a processing system having at least one hardware processor, perform operations comprising:
receiving, by a storage appliance, a request to restore a file to the storage appliance, the file having been previously segmented into a plurality of deduplicated segments, each deduplicated segment having a fingerprint;
determining, by the storage appliance, a tier of storage where all deduplicated segments of the requested file are stored;
for each batch of fingerprints of a plurality of batches of fingerprints, iteratively performing:
determining a hit-rate as a first ratio of segments associated with a previous batch of fingerprints that are stored on the storage appliance vs. stored on the tier storage;
retrieving, from the storage appliance, the batch of fingerprints, each fingerprint in the batch of fingerprints identifying a deduplicated segment of the file;
for each fingerprint in the batch of fingerprints, accessing an index of the storage appliance to determine where the segment associated with the fingerprint is stored;
determining whether an advanced direct restore (ADR) was enabled or disabled for the previous batch of fingerprints;
in response to determining that the ADR was disabled for the previous batch of fingerprints, determining the hit-rate as a second ratio of segments associated with a portion of fingerprints of the batch of fingerprints that are stored on the storage appliance vs. stored on the tier storage;
determining whether the hit-rate is below a threshold;
in response to determining that the hit-rate is above the threshold, enabling the ADR for the batch of fingerprints and, prefetching segments associated with the batch of fingerprints by retrieving at least one segment associated with a fingerprint in the batch of fingerprints from the storage appliance and retrieving at least one other segment associated with one other fingerprint in the batch of fingerprints from the tier of storage where all segments of the file are stored; and
in response to determining that the hit-rate is below the threshold, disabling the ADR and retrieving all of the segments associated with the batch of fingerprints from the tier of storage.
8. The medium of claim 7 , wherein the tier of storage where all deduplicated segments of the file are located is distinct from the storage appliance.
9. The medium of claim 7 , wherein all segments of the file are located on a remote tier of storage that comprises a cloud storage service.
10. The medium of claim 7 , wherein the request is received from an application in the context of a storage appliance input/output (I/O) operation from the application.
11. The medium of claim 7 , wherein the prefetched segments are read and provided to an application that requested that the file be restored.
12. The medium of claim 7 , wherein the batch of fingerprints are ordered in a file order of the prefetched segments.
13. A system comprising:
a processing system having at least one hardware processor, the processing system coupled to a memory programmed with executable instructions that, when executed by the processing system, perform operations comprising:
receiving, by a storage appliance, a request to restore a file to the storage appliance, the file having been previously segmented into a plurality of deduplicated segments, each deduplicated segment having a fingerprint;
determining, by the storage appliance, a tier of storage where all deduplicated segments of the requested file are stored;
for each batch of fingerprints of a plurality of fingerprints, iteratively performing:
determining, a hit-rate as a first ratio of segments associated with a previous batch of fingerprints that are stored on the storage appliance vs. stored on the tier storage;
retrieving, from the storage appliance, the batch of fingerprints, each fingerprint in the batch of fingerprints identifying a deduplicated segment of the file;
for each fingerprint in the batch of fingerprints, accessing an index of the storage appliance to determine where the segment associated with the fingerprint is stored;
determining whether an advanced direct restore (ADR) was enabled or disabled for the previous batch of fingerprints;
in response to determining that the ADR was disabled for the previous batch of fingerprints, determining the hit-rate as a second ratio of segments associated with a portion of the fingerprints of the batch of fingerprints that are stored on the storage appliance vs. stored on the tier storage;
determining whether the hit-rate is below a threshold;
in response to determining that the hit-rate is above the threshold, enabling the ADR for the batch of fingerprints and prefetching segments associated with the batch of fingerprints by retrieving at least one segment associated with a fingerprint in the batch of fingerprints from the storage appliance and retrieving at least one other segment associated with one other fingerprint in the plurality of fingerprints from the tier of storage where all segments of the file are stored; and
in response to determining that the hit-rate is below the threshold, disabling the ADR and retrieving all of the segments associated with the batch of fingerprints from the tier storage.
14. The system of claim 13 , wherein the tier of storage where all deduplicated segments of the file are located is distinct from the storage appliance.
15. The system of claim 13 , wherein all segments of the file are located on a remote tier of storage that comprises a cloud storage service.
16. The system of claim 13 , wherein the request is received from an application in the context of a storage appliance input/output (I/O) operation from the application.
17. The system of claim 13 , wherein the prefetched segments are read and provided to an application that requested that the file be restored.
18. The system of claim 13 , wherein the batch of fingerprints are ordered in a file order of the prefetched segments.