Multi-threaded dynamic per-file read-ahead cache for deduplication system
A multi-stream restore method for reducing restore time of file data to service an external read request for file data stored by a data deduplication system employs a per-file read-ahead cache that is populated by multiple parallel internal read-ahead streams. The number of parallel read-ahead streams is dynamically variable based upon processing conditions and a set of heuristically pre-determined criteria that characterize the file data including a data access pattern that is sequential and a processing load that is below a predetermined threshold. Processing conditions are periodically monitored and adjustments to the multi-stream data access are made to optimize performance.
1. A method of improving access time to provide quicker restores in accessing data in a file in a storage layer of a data deduplication system to service an external read request for a data restore, comprising:
confirming that access of said requested data is sequential;
in response to said confirming, opening multiple internal read-ahead streams that read ahead in parallel data in increments of a chunk of said file in said storage layer to prefetch said data;
dynamically varying, to optimize processing and speed of said data access, the number of said multiple read-ahead streams that read said data based upon the present system processing conditions and based upon a set of preselected criteria applicable to characteristics of the file being read;
populating a cache with the data read by said multiple read-ahead streams; and
servicing said external read request with the data populated in the cache.
2. The method of claim 1 , wherein said confirming further comprises confirming that the size of said file being accessed is greater than a predetermined threshold size prior to opening said multiple read-ahead streams.
3. The method of claim 2 , wherein said predetermined threshold size is a size below which the costs of setting up said multiple read-ahead stream access exceed the performance benefit obtained from multiple read-ahead stream access.
4. The method of claim 1 , wherein said preselected criteria include said file data being stored in a storage tier having an input/output (I/O) bandwidth that is smaller than one at which the processing resource costs required for multiple stream reading exceed the performance benefit obtained.
5. The method of claim 1 , wherein said present system processing conditions comprise a system processing load below a predetermined threshold load corresponding to said optimum processing.
6. Then method of claim 1 , wherein said dynamically varying comprises monitoring periodically system processing conditions including one or more of a number of reads and writes, the central processing unit (CPU) usage, and disk or input/output (I/O) responses.
7. The method of claim 6 , wherein said dynamically varying comprises changing the number of internal multiple streams accessing data between a single stream and a number of multiple streams to maintain said system processing conditions within predetermined limits.
8. The method of claim 1 , wherein said dynamically varying comprises periodically testing for compliance of said processing conditions and said preselected criteria with predetermined thresholds, and upon said processing conditions or one or more of said preselected criteria failing to satisfy said predetermined thresholds, backing off said multiple stream reading, and setting a retry time interval before re-testing for compliance with said predetermined thresholds to renew multiple stream reading.
9. The method of claim 8 , wherein said setting a retry time interval comprises setting a back-off factor, n, and a retry time unit, t, and setting said retry time as n*t, where said back-off factor, n, is an integer that increments each time a retry fails to satisfy said predetermined thresholds, and the retry time unit, t, is a fixed unit of time.
10. A non-transitory computer readable storage medium embodying executable instructions for controlling a processor of a storage deduplication system to perform a method of optimizing access time to provide quicker restores in accessing data in a file stored in said system in response to an external read request, comprising:
confirming that access of said requested data is sequential;
in response to said confirming, opening multiple internal read-ahead streams that read ahead in parallel data in increments of a chunk of said file in said storage layer to prefetch said data;
dynamically varying, to optimize processing and speed of said data access, the number of said multiple read-ahead streams that read said data based upon the present system processing conditions and based upon a set of preselected criteria applicable to characteristics of the file being read;
populating a cache with the data read by said multiple read-ahead streams; and
servicing said external read request with the data populated in the cache.
11. The non-transitory computer readable storage medium of claim 10 , wherein said confirming further comprises confirming that the size of said file being accessed is greater than a predetermined threshold size prior to opening said multiple read-ahead streams.
12. The non-transitory computer readable storage medium of claim 11 , wherein said predetermined threshold size is a size below which the costs of setting up said multiple read-ahead stream access exceed the performance benefit obtained from multiple read-ahead stream access.
13. The non-transitory computer readable storage medium of claim 10 , wherein said preselected criteria include said file data being stored in a storage tier having an input/output (I/O) bandwidth that is smaller than one at which the processing resource costs required for multiple stream reading exceed the performance benefit obtained.
14. The non-transitory computer readable storage medium of claim 10 , wherein said present system processing conditions comprise a system processing load below a predetermined threshold load corresponding to said optimum processing.
15. The non-transitory computer readable storage medium of claim 10 , wherein said dynamically varying comprises monitoring periodically system processing conditions including one or more of a number of reads and writes, the central processing (CPU) usage, and disk or input/output (I/O) responses.
16. The non-transitory computer readable storage medium of claim 15 , wherein said dynamically varying comprises changing the number of internal multiple streams accessing data between a single stream and a number of multiple streams to maintain said system processing conditions within predetermined limits.
17. The non-transitory computer readable storage medium of claim 10 , wherein said dynamically varying comprises periodically testing for compliance of said processing conditions and said preselected criteria with predetermined thresholds, and upon said processing conditions or one or more of said preselected criteria failing to satisfy said predetermined thresholds, backing off said multiple stream reading, and setting a retry time interval before re-testing for compliance with said predetermined thresholds to renew multiple stream reading.
18. The non-transitory computer readable storage medium of claim 17 , wherein said setting a retry time interval comprises setting a back-off factor, n, and a retry time unit, t, and setting said retry time as n*t, where said back-off factor, n, is an integer that increments each time a retry fails to satisfy said predetermined thresholds, and the retry time unit, t, is a fixed unit of time.