Systems and methods for statistical caching
The present invention relates to systems and methods for statistical caching. Inputs are captured via an appropriate network protocol. The input includes statistical data and a corresponding cache key. The values for each cache key within a cache are compacted using the input. The compacting involves determining if the corresponding cache key is already set within the cache, and if the cache key is present, aggregating the statistical data with the value stored within the cache to generate an updated value. The updated cache may be periodically synchronized with a final data store. Additionally, each operation performed by the statistical cache may be recorded in a transaction log for fault tolerance.
1. A computer implemented method for statistical caching of input events to improve computer operation when receiving large numbers of inputs comprising:
capturing a plurality of inputs via appropriate network protocols, wherein the inputs each have statistical data and a corresponding cache key:
comparing the corresponding cache key for each of the captured plurality of inputs against stored keys;
storing the statistical data of a first subset of inputs, with corresponding cache keys, to a cache memory when the corresponding cache key is not found in the stored keys;
retrieving information already present within the cache memory and compacting the statistical data of a second subset of inputs, with corresponding cache keys, with the retrieved information already present within the cache memory to generate an aggregate statistical variable data when the corresponding cache key is found in the stored keys, and updating the stored keys identifying the aggregate statistical variable;
synchronizing the cache memory with a final data store;
recording each operation in a transaction log; and
regenerating the cache by replaying the transaction log in the event of a disruption.
2. The method of claim 1 , wherein the appropriate network protocol is selected by desired throughput and data modeling.
3. The method of claim 1 , wherein the synchronization is periodic.
4. The method of claim 3 , wherein the synchronization occurs after a set period of time.
5. The method of claim 3 , wherein the synchronization occurs after a set number of inputs.
6. The method of claim 1 , wherein the cache key is a string, and wherein the value is a sequence of bytes.
7. A statistical caching system for managing the storage of input events comprising:
a data capture interface for capturing a plurality of inputs via appropriate network protocols, wherein the inputs each have statistical data and a corresponding cache key;
a compacting module comprising a processor and a cache memory, configured to:
compare the corresponding cache key for each of the captured plurality of inputs against stored keys;
store the statistical data of a first subset of inputs, with corresponding cache keys, to a cache memory when the corresponding cache key is not found in the stored keys;
retrieve information already present within the cache memory and compact the statistical data of a second subset of inputs, with corresponding cache keys, with the retrieved information already present within the cache memory to generate an aggregate statistical variable when the corresponding cache key is found in the stored keys, and updating the stored keys identifying the aggregate statistical variable;
record each operation in a transaction log; and
regenerate the cache by replaying the transaction log in the event of a disruption; and
a persisting module comprising a processor and coupled to a bus for synchronizing the cache memory with a final data store.
8. The system of claim 7 , wherein the data capture module selects an appropriate network protocol based upon desired throughput and data modeling.
9. The system of claim 7 , wherein the persisting module performs the synchronization periodically.
10. The system of claim 9 , wherein the persisting module performs the synchronization after a set period of time.
11. The system of claim 9 , wherein the persisting module performs the synchronization after a set number of inputs.
12. The system of claim 7 , wherein the cache key is a string, and wherein the value is a sequence of bytes.