EFFICIENT AGGREGATION, STORAGE AND QUERYING OF LARGE VOLUME METRICS
The present system provide for more efficient processing, storage and querying of metrics from a distributed system from which large volumes of metrics are collected. The present metrics processing system may store billions of performance metrics in a persistence storage system, such as an HBase storage system, for several days, with minimum space required and at the same time retaining a low level data granularity. The reporting queries may use a unique technique to find required metrics in the HBase persistence store using a portion of the key as a bit array. The present metrics processing system may user a very small number of keys to store minute level metrics data for a metric for several hours. The metric values may be pivoted to multiple time-bucketed keys at different times during their life time in the system.
1 . A method for processing metrics, comprising:
receiving a plurality of payloads which each include time series data;
storing a first time series data associated with a first time range with a first key; and
storing the first time series and at least one other time series data of the plurality of time series data associated with a second time range with a second key.
2 . The method of claim 1 , further comprising generating a bit map for the time series data stored with the second key.
3 . The method of claim 1 , further comprising storing the time series data in a byte array associated with the first key and a time the data was received.
4 . The method of claim 3 , further comprising storing a plurality of byte arrays associated with first key as a single byte array associated with the second key.
5 . The method of claim 1 , further comprising storing the time series data associated with the second key as a compressed byte array associated with a third key.
6 . The method of claim 1 , further comprising storing a plurality of bit maps associated with the second key as a single bit map associated with the third key.
7 . The method of claim 1 , wherein the first time series and at least one other time series data of the plurality of time series data is stored with the second key after a threshold period of time has been satisfied.
8 . The method of claim 1 , further comprising allowing the time series data associated with the first key to be overwritten once the time series data associated with the first key is associated with the second key.
9 . A method for processing metrics, comprising:
receiving metric data for a metric type;
updating one or more groups of data associated with a time period for the metric type, wherein the groups are associated with at least two different periods of time; and
providing at least two groups associated with different periods of time in response to a query for metric data over a period of time.
10 . The method according to claim 9 , wherein the groups of data are provided to a cache for receiving queries.
11 . A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for processing metrics, the method comprising:
receiving a plurality of payloads which each include time series data;
storing a first time series data associated with a first time range with a first key; and
storing the first time series and at least one other time series data of the plurality of time series data associated with a second time range with a second key.
12 . A system for processing metrics, comprising:
a processor;
a memory; and
one or more modules stored in memory and executable by a processor to receive a plurality of payloads which each include time series data, store a first time series data associated with a first time range with a first key, and store the first time series and at least one other time series data of the plurality of time series data associated with a second time range with a second key.