Aggregation and routing of metric time series data
Described herein are techniques for aggregating metric time series data upon ingestion, such as prior to intake, analysis, or processing by a data quantizer system. The disclosed techniques can transforming metric time series data into aggregated metric time series data, optionally with reduced complexity or dimensionality, while retaining various statistical information characteristic of the original source metric time series data. The aggregated metric time series data can be used in the same way as the original source metric time series data or other metric time series data, such as for processing or generating streaming analytics data or triggering events or notifications by a data quantizer system.
1 . A computer-implemented method, comprising:
receiving a plurality of metric time series at a computing system;
identifying, using one or more filter criteria, a particular metric time series of the plurality of metric time series for aggregation, the particular metric time series including a plurality of dimensions;
performing, using one or more aggregation criteria, aggregation of data points of the particular metric time series to generate an aggregated metric time series, wherein the aggregated metric time series includes fewer dimensions than the plurality of dimensions included in the particular metric time series;
routing the particular metric time series to a data store;
routing the aggregated metric time series to a data quantizer system;
generating, by the data quantizer system, roll-up data points based on the aggregated metric time series, the roll-up data points including quantized values associated with a plurality of time intervals, wherein the aggregated metric time series includes, for each time interval of the plurality of time intervals and for each value of a plurality of values of a particular dimension included in the aggregated metric time series, an aggregated data point comprising a plurality of data values, and wherein the plurality of data values includes a first data value that is computed as a sum of differences between (i) metric values of data points included in the particular metric time series and associated with a value of the particular dimension and belonging to a time interval and (ii) metric values of data points included in the particular metric time series and associated with the value of the particular dimension and belonging to an immediately previous time interval; and
processing, by the computing system, the roll-up data points generated over the plurality of time intervals to determine an overall usage of a plurality of processor cores distributed in one or more datacenters, wherein each of the plurality of metric time series includes information related to processor usage in a data center of the one or more datacenters.
2 . The computer-implemented method of claim 1 , further comprising routing other metric time series from the plurality of metric time series that are not identified for aggregation to one or more of a data quantizer system, a data store, or a remote system.
3 . The computer-implemented method of claim 1 , wherein each metric time series in the plurality of metric time series includes one or more dimensions and/or one or more metadata values.
4 . The computer-implemented method of claim 1 , wherein each metric time series in the plurality of metric time series comprises a plurality of data points, each data point including a timestamp, one or more dimensions, and one or more values.
5 . The computer-implemented method of claim 1 , wherein the aggregated metric time series comprises a plurality of data points, each data point including a timestamp, one or more dimensions, and a plurality of values.
6 . The computer-implemented method of claim 1 , wherein the one or more filter criteria identify the particular dimension or metadata value and wherein identifying the particular metric time series comprises matching the particular dimension or metadata value to the particular metric time series.
7 . The computer-implemented method of claim 1 , further comprising obtaining the one or more filter criteria from a remote system or receiving input corresponding to the one or more filter criteria.
8 . The computer-implemented method of claim 1 , wherein the aggregation criteria specifies a number of data points to consolidate for each aggregated data point.
9 . The computer-implemented method of claim 1 , wherein the aggregation criteria specifies one or more dimensions included in the particular metric time series to retain for each aggregated data point or to drop for each aggregated data point.
10 . The computer-implemented method of claim 1 , wherein the aggregation criteria specifies a computation algorithm for aggregating values for each aggregated data point.
11 . The computer-implemented method of claim 1 , wherein each of multiple data points of the particular metric time series includes a corresponding value and wherein aggregating comprises creating aggregated data points using a plurality of the corresponding values from different data points of the multiple data points.
12 . The computer-implemented method of claim 1 , wherein data points of the particular metric time series include first dimension values for a first dimension, second dimension values for a second dimension, and a metric value, wherein aggregation includes generating, from a plurality of data points of the particular metric time series, an aggregated data point, wherein the aggregated data point includes a plurality of entries, each entry corresponding to a different first dimension value for the first dimension.
13 . The computer-implemented method of claim 12 , wherein the plurality of data values further includes:
a count of a number of data points including a particular dimension value,
a sum of metric values from data points including the particular dimension value,
a maximum of metric values from data points including the particular dimension value,
a minimum of metric values from data points including the particular dimension value, and
an average of metric values from data points including the particular dimension value.
14 . The computer-implemented method of claim 1 , further comprising obtaining the one or more aggregation criteria from a remote system or receiving input corresponding to the one or more aggregation criteria.
15 . The computer-implemented method of claim 1 , further comprising:
determining, based on one or more routing criteria, whether the particular metric time series is to be communicated to a remote system.
16 . The computer-implemented method of claim 15 , wherein the one or more routing criteria indicates that the aggregated metric time series is to be communicated to a remote system.
17 . The computer-implemented method of claim 15 , further comprising obtaining the one or more routing criteria from a remote system or receiving input corresponding to the one or more routing criteria.
18 . A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving a plurality of metric time series at a computing system;
identifying, using one or more filter criteria, a particular metric time series of the plurality of metric time series for aggregation, the particular metric time series including a plurality of dimensions;
performing, using one or more aggregation criteria, aggregation of data points of the particular metric time series to generate an aggregated metric time series, wherein the aggregated metric time series includes fewer dimensions than the plurality of dimensions included in the particular metric time series;
routing the particular metric time series to a data store;
routing the aggregated metric time series to a data quantizer system;
generating, by the data quantizer system, roll-up data points based on the aggregated metric time series, the roll-up data points including quantized values associated with a plurality of time intervals, wherein the aggregated metric time series includes, for each time interval of the plurality of time intervals and for each value of a plurality of values of a particular dimension included in the aggregated metric time series, an aggregated data point comprising a plurality of data values, and wherein the plurality of data values includes a first data value that is computed as a sum of differences between (i) metric values of data points included in the particular metric time series and associated with a value of the particular dimension and belonging to a time interval and (ii) metric values of data points included in the particular metric time series and associated with the value of the particular dimension and belonging to an immediately previous time interval; and
processing the roll-up data points generated over the plurality of time intervals to determine an overall usage of a plurality of processor cores distributed in one or more datacenters, wherein each of the plurality of metric time series includes information related to processor usage in a data center of the one or more datacenters.
19 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a plurality of metric time series at a computing system;
identifying, using one or more filter criteria, a particular metric time series of the plurality of metric time series for aggregation, the particular metric time series including a plurality of dimensions;
performing, using one or more aggregation criteria, aggregation of data points of the particular metric time series to generate an aggregated metric time series, wherein the aggregated metric time series includes fewer dimensions than the plurality of dimensions included in the particular metric time series;
routing the particular metric time series to a data store;
routing the aggregated metric time series to a data quantizer system; and
generating, by the data quantizer system, roll-up data points based on the aggregated metric time series, the roll-up data points including quantized values associated with a plurality of time intervals, wherein the aggregated metric time series includes, for each time interval of the plurality of time intervals and for each value of a plurality of values of a particular dimension included in the aggregated metric time series, an aggregated data point comprising a plurality of data values, and wherein the plurality of data values includes a first data value that is computed as a sum of differences between (i) metric values of data points included in the particular metric time series and associated with a value of the particular dimension and belonging to a time interval and (ii) metric values of data points included in the particular metric time series and associated with the value of the particular dimension and belonging to an immediately previous time interval; and
processing, by the computing system, the roll-up data points generated over the plurality of time intervals to determine an overall usage of a plurality of processor cores distributed in one or more datacenters, wherein each of the plurality of metric time series includes information related to processor usage in a data center of the one or more datacenters.