Methods and systems that efficiently store metric data
The current document is directed to methods and systems that collect metric data within computing facilities, including large data centers and cloud-computing facilities. In a described implementation, lower and higher metric-data-value thresholds are used to partition collected metric data into outlying metric data and inlying metric data. The inlying metric data is quantized to compress the inlying metric data and adjacent data points having the same quantized metric-data values are eliminated, to further compress the inlying metric data. The resulting compressed data includes original metric-data representations for outlier data points and compressed metric-data representations for inlier data points, providing accurate restored metric-data values for significant data points when compressed metric data is decompressed.
1. A metric-data storage subsystem within a distributed computer system, the metric-data collection-and-storage subsystem comprising:
one or more processors;
one or more memories;
one or more data-storage devices;
one or more virtual machines instantiated by computer instructions stored in one or more of the one or more memories and executed by one or more of the one or more processors that together decompress stored metric data by
receiving a reference to a container containing compressed metric data points,
extracting compression parameters from the container, and
while unextracted metric data points remain in the container,
determining a next time in a time sequence of metric data points generated by decompressing the stored metric data,
when an unextracted outlier metric data point corresponds to the next time,
extracting the outlier metric data point, and
appending the extracted metric data point to the time sequence of metric data points,
when an unextracted compressed inlier metric data point corresponds to the next time,
extracting the compressed inlier metric data point,
decompressing the compressed inlier metric data point, and
appending the decompressing the inlier metric data point to the time sequence of metric data points, and
when no unextracted inlier metric data point corresponds to the next time, appending a copy of the most recently decompressed inlier metric data point to the time sequence of metric data points.
2. The metric-data storage subsystem of claim 1 wherein each received metric data point is representable as a timestamp/metric-data-value pair.
3. The metric-data storage subsystem of claim 2 wherein outlier metric data points include:
lower-threshold-outlier metric data points with metric-data values less than a lowest-valued quantization threshold metric-data value; and
higher-threshold-outlier metric data points with metric-data values greater than a highest-valued quantization threshold metric-data value.
4. The metric-data storage subsystem of claim 3 wherein the extracted compression parameters include:
a number n that indicates a number of representative quantization-interval metric-data values;
the representative quantization-interval metric-data values;
an indication of a number of lower-threshold-outlier metric data points;
an indication of a number of higher-threshold-outlier metric data points; and
an indication of a number of stored inlier metric data points.
5. The metric-data storage subsystem of claim 3 wherein decompressing a compressed inlier metric data point comprises:
using a quantization-interval identifier included in the compressed inlier metric data point to replace the quantization-interval identifier with a corresponding representative quantization-interval metric-data value for the quantization interval identified by the quantization-interval identifier.