Dynamic caching module selection for optimized data deduplication
Embodiments of the invention provide a method, system and computer program product for dynamic caching module selection for optimized data deduplication. In an embodiment of the invention, a method for dynamic caching module selection for optimized data deduplication is provided. The method includes receiving a request to retrieve data and classifying the request. The method also includes identifying from amongst multiple different caching modules each with a different configuration a particular caching module associated with the classification of the request. Finally, the method includes deduplicating the data in the identified caching module.
1. A data deduplication data processing system configured for dynamic caching module selection for optimized data deduplication, the system comprising:
a server communicatively coupled to a data store and a plurality of client computers over a computer communications network;
middleware disposed between the server and the client computers and executing in memory of a host computer, the middleware comprising a plurality of different caching modules, each of the different caching modules having a different configuration and ones of the plurality of the different caching modules comprising program code enabled to deduplicate data in an identified particular one of the different caching modules; and,
a scheduler comprising program code executing in memory of a host computer and enabled to classify a request to retrieve data from the data store of the server, to identify from amongst the different caching modules a particular caching module associated with the classification of the request, and to route the data for deduplication in the identified particular caching module, but bypassing the deduplication of the data when the request indicates that data is encrypted.
2. The system of claim 1 , wherein the request is classified according to a table correlating different requests for different ones of the caching modules.
3. The system of claim 1 , wherein the caching modules include byte caching modules each byte caching module configured with a different fingerprint size.
4. The system of claim 3 , wherein the byte caching modules additionally include an object caching module.
5. The system of claim 4 , wherein the object caching module compresses objects.
6. The system of claim 2 , wherein the table correlates a protocol and network address for each of the different requests with a corresponding one of the caching modules.
7. The system of claim 1 , wherein the program code of the scheduler bypasses deduplication of the data responsive to an indication in the scheduler to bypass deduplication of the data referenced by the request.
8. The system of claim 1 , further comprising a table, wherein the table includes entries determined by processing training data for different requests in each of the different caching modules and correlates the request with an optimal one of the modules.
9. The system of claim 8 , wherein the table is constructed by:
submitting the training data to each of the modules from different servers according to different protocols,
monitoring performance metrics of each of the modules and measuring the performance for throughput, processor and memory utilization and response time,
submitting the measured metrics to a performance function that weights different metrics for utilization of different resources and then sums the weighted metrics into an aggregated metric,
comparing the aggregate metric for a particular training data set against other aggregate metrics for the training data set in different ones of the modules, and
selecting an optimal one of the modules corresponding to a classification for the training data.