ROLLING UPGRADE OF METRIC COLLECTION AND AGGREGATION SYSTEM
A system processes a large volume of real time hierarchical system metrics using distributed computing by stateless processes. The metrics processing system receives different types of hierarchical metrics coming from different sources and then aggregates the metrics by their hierarchy. The system is on-demand, cloud based, multi-tenant and highly available. The system makes the aggregated metrics available for reporting and policy triggers in real time. The aggregators and collectors may be upgraded to new versions with minimal loss in data.
1 . A method for processing metrics, comprising:
transmitting by a collector module stored in memory and executed by a processor a data set to one of a first group of aggregators;
providing a plurality of updated aggregators concurrently with the first group of aggregators;
transmitting subsequent data sets to one of the aggregators in the first group of aggregators or an aggregator of the updated aggregators during a first time period; and
transmitting subsequent data sets to the updated aggregators during a second time period, the second time period occurring after the first time period.
2 . The method of claim 1 , wherein the data sets are provided to a particular aggregator based on a hash of the data set.
3 . The method of claim 1 , further comprising:
receiving a payload which includes sets of data; and
aggregating received metrics by each of a plurality of aggregators;
4 . The method of claim 1 , the updated aggregators associated with a start time, the subsequent data sets associated with a time stamp after the start time being transmitted to an aggregator of the updated aggregators.
5 . The method of claim 1 , wherein data sets received after the start time and having a time stamp within the first time period being transmitted to one of the aggregators in the first group of aggregators.
6 . The method of claim 1 , wherein data sets received after the start time and having a time stamp after the first time period are not transmitted to the first group of aggregators or the updated aggregators.
7 . The method of claim 1 , wherein the data sets are received from one or more collectors, the collectors receiving the start time and addresses of the updated aggregators from memory.
8 . The method of claim 1 , wherein the first group of aggregators is associated with a first version of aggregators and the updated aggregators are associated with a second version of aggregators.
9 . A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for processing metrics, the method comprising:
transmitting a data set to one of a first group of aggregators;
providing a plurality of updated aggregators concurrently with the first group of aggregators;
transmitting subsequent data sets to one of the aggregators in the first group of aggregators or an aggregator of the updated aggregators during a first time period; and
transmitting subsequent data sets to the updated aggregators during a second time period, the second time period occurring after the first time period.
10 . The non-transitory computer readable storage medium of claim 9 , wherein the data sets are provided to a particular aggregator based on a hash of the data set.
11 . The non-transitory computer readable storage medium of claim 9 , further comprising:
receiving a payload which includes sets of data;
aggregating received metrics by each of a plurality of aggregators.
12 . The non-transitory computer readable storage medium of claim 9 , the updated aggregators associated with a start time, the subsequent data sets associated with a time stamp after the start time being transmitted to an aggregator of the updated aggregators.
13 . The non-transitory computer readable storage medium of claim 9 , wherein data sets received after the start time and having a time stamp within the first time period being transmitted to one of the aggregators in the first group of aggregators.
14 . The non-transitory computer readable storage medium of claim 9 , wherein data sets received after the start time and having a time stamp after the first time period are not transmitted to the first group of aggregators or the updated aggregators.
15 . The non-transitory computer readable storage medium of claim 9 , wherein the data sets are received from one or more collectors, the collectors receiving the start time and addresses of the updated aggregators from memory.
16 . The non-transitory computer readable storage medium of claim 9 , wherein the first group of aggregators is associated with a first version of aggregators and the updated aggregators are associated with a second version of aggregators.
17 . A system for processing metrics, comprising:
a processor;
a memory; and
one or more modules stored in memory and executable by a processor to transmit a data set to one of a first group of aggregators, provide a plurality of updated aggregators concurrently with the first group of aggregators, transmit subsequent data sets to one of the aggregators in the first group of aggregators or an aggregator of the updated aggregators during a first time period, and transmit subsequent data sets to the updated aggregators during a second time period, the second time period occurring after the first time period.
18 . The system of claim 17 , wherein the data sets are provided to a particular aggregator based on a hash of the data set.
19 . The system of claim 17 , the one or more modules further executable to receive a payload which includes sets of data and aggregate received metrics by each of a plurality of aggregators;
20 . The system of claim 17 , the updated aggregators associated with a start time, the subsequent data sets associated with a time stamp after the start time being transmitted to an aggregator of the updated aggregators.
21 . The system of claim 17 , wherein data sets received after the start time and having a time stamp within the first time period being transmitted to one of the aggregators in the first group of aggregators.
22 . The system of claim 17 , wherein data sets received after the start time and having a time stamp after the first time period are not transmitted to the first group of aggregators or the updated aggregators.
23 . The system of claim 17 , wherein the data sets are received from one or more collectors, the collectors receiving the start time and addresses of the updated aggregators from memory.
24 . The system of claim 17 , wherein the first group of aggregators is associated with a first version of aggregators and the updated aggregators are associated with a second version of aggregators.