Self-optimizing caching system and method for data records
A system and appertaining method provide for pre-fetching records from a central data base to a local storage area in order to reduce delays associated with the data transfers. User patterns for requesting data records are analyzed and rules/strategies are generated that permit an optimal pre-fetching of the records based on the user patterns. The rules/strategies are implemented by a routine that pre-fetches the data records so that users have the records available to them when needed.
1. A method for pre-fetching data records from a central data store to a local data store, comprising the steps of:
populating the central data store with records from a record source;
making multiple requests by each of a plurality of respective users with each user having its own respective proxy service for downloading the records from the central data store to the local data store of the respective user by a record data management module via the respective proxy service so that the respective user is not directly connected to the record data management module, said respective proxy service outputting metadata defining who accesses what data, where, and when, and wherein each request has a request vector associated with it defining a plurality of characteristics of the request;
aggregating said metadata from all of said proxy services relating to all of said multiple requests from the respective users into a separate metadata repository which is separate from said users, central data store and said record data management module, and wherein said aggregating is initiated in an event driven manner by the proxy service in a poled manner by a routine associated with the metadata repository;
translating the metadata into dynamically created pre-fetch rules and strategies based on the respective user metadata and using the request vectors for quantification of relationships between requests, wherein similarities and regularities in requests are identified by grouping similar, close-by objects into clusters, using a density based algorithm for growing the clusters that are extended as long as a density of objects in a corresponding neighborhood exceeds a predefined threshold, and wherein the density based algorithm as well as a hybrid clustering which uses a combination of the request vector quantification and the density based algorithm is employed;
initiating by the record data management module automatic downloading of pre-fetched records from the central data store to the local data store of the respective user according to the pre-fetch rules and strategies; and
updating the pre-fetch rules and strategies based on additional requests by the respective user for downloading the records from the central data store to the local data store.
2. The method according to claim 1 , further comprising:
automatically downloading pre-fetched records to more than one local data store associated with the respective user according to the pre-fetch rules and strategies.
3. The method according to claim 1 , further comprising:
sending the requests by the proxy service to the record data management module after receiving the requests.
4. The method according to claim 1 , further comprising:
sending the requests to the proxy services and the record data management module in parallel.
5. The method according to claim 1 , further comprising segregating, by the proxy services, the metadata from the request and locally storing this metadata.
6. The method according to claim 1 , wherein the translating of the metadata into the pre-fetch rules and strategies further utilizes external information not contained within the metadata itself.
7. The method according to claim 6 , wherein the external information is selected from the group consisting of central data store availability and utilization, network bandwidth, and user schedules.
8. The method according to claim 1 , further comprising:
locally storing the records in a cache of the proxy service.
9. The method according to claim 1 , further comprising:
caching the requested records in the metadata repository or the record data management module.
10. The method according to claim 9 , wherein the metadata comprises a requesting user ID, a location originating the request, a workstation originating the request, and a time of the request.
11. The method according to claim 1 , further comprising:
replicating and synchronizing the records for the purpose of error recovery.
12. The method according to claim 1 , wherein the records comprise medical image data.
13. The method according to claim 12 , wherein the record source is a medical imaging apparatus.
14. The method according to claim 1 , wherein the central data store is implemented with a Picture Archiving and Communication System (PACS).
15. The method according to claim 1 , wherein the local data store is selected from the group consisting of a random access memory, a hard drive, a local file system or a local database and is a local data store on a client or another server.
16. The method according to claim 1 , wherein the translating of the metadata into the pre-fetch rules and strategies comprises:
creating said clusters of similar requests;
for every cluster, extracting a set of properties that caused the clustering; and
for every set of extracted properties, generating and storing the pre-fetch strategy.
17. The method according to claim 16 , further comprising utilizing a hierarchical clustering algorithm that either: a) stepwise aggregates objects into groups in an agglomerative approach, or b) stepwise splits or subdivides bigger groups into subgroups in a partioning approach using said partioning algorithm.