COLLECTION AND AGGREGATION OF LARGE VOLUME OF METRICS
A system processes a large volume of real time hierarchical system metrics using distributed computing by stateless processes. The metrics processing system receives different types of hierarchical metrics coming from different sources and then aggregates the metrics by their hierarchy. The system is on-demand, cloud based, multitenant and highly available. The system makes the aggregated metrics available for reporting and policy triggers in real time. The metrics aggregation and roll up method involves two different classes of stateless java programs that work in tandem to receive, aggregate and roll up the incoming metrics. The method also involves a persistence store to save incoming and aggregated metrics. The method uses a quorum to communicate between these processes.
1 . A method for processing metrics, comprising:
receiving a payload which includes sets of data;
generating a hash from each set of data;
transmitting each data set to one of a plurality of aggregators based on the hash; and
aggregating received metrics by each of a plurality of aggregators.
2 . The method of claim 1 , wherein each of the plurality of aggregators receive metrics associated with a particular hash value.
3 . The method of claim 1 , wherein the hash is generated from a metric identifier.
4 . The method of claim 1 , wherein the same hash function is applied by a plurality of collectors.
5 . The method of claim 1 , further comprising storing the data set at a remote store by each of one or more collectors, wherein each data set is received by a collector.
6 . The method of claim 5 , further comprising analyzing previous payload processing by a selected collector of the plurality of collectors.
7 . The method of claim 1 , further comprising providing the aggregated data to a cache.
8 . The method of claim 1 , further comprising:
detecting that an aggregator has failed; and
adding an aggregator to the plurality of aggregators.
9 . The method of claim 8 , further comprising retrieving historic metric processing data for metrics associated with the added aggregator.
10 . The method of claim 8 , further comprising:
transmitting selected metric data to the added aggregator; and
transmitting the selected metric data to the a second aggregator of the plurality of aggregators.
11 . A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for processing metrics, the method comprising:
receiving a payload which includes sets of data;
generating a hash from each set of data;
transmitting each data set to one of a plurality of aggregators based on the hash; and
aggregating received metrics by each of a plurality of aggregators.
12 . A system for processing metrics, comprising:
a processor;
a memory; and
one or more modules stored in memory and executable by a processor to receive a payload which includes sets of data, generate a hash from each set of data, transmit each data set to one of a plurality of aggregators based on the hash, and aggregate received metrics by each of a plurality of aggregators.