Cardinality of time series
Querying of time-aware metrics time series includes receiving a query, the query comprising a set of query metadata and a query time range. It further includes, based at least in part on the set of query metadata and the query time range, selecting a time series from a plurality of metrics time series. Each metrics time series in the plurality of metrics time series is associated with a set of metadata and an active interval of time. A set of metadata associated with the selected time series matches the set of query metadata, and an active interval of time associated with the selected metrics time series intersects with the query time range. The selected metrics time series is returned.
1. A computer-implemented method comprising:
providing, by one or more processors, a plurality of time series, each time series comprising a plurality of values of a metric and a corresponding timestamp for each value of the metric, the plurality of values being indexed in order of the timestamps;
for each time series, maintaining, by the one or more processors, time-series metadata stored separately from computer storage where the time series is stored, the time-series metadata comprising a value of a newest timestamp corresponding to a newest metric value stored in the time series, the newest timestamp being the timestamp stored last in the time series;
receiving, by the one or more processors, a query, the query comprising a time selector for selecting entries from the plurality of time series within a time period;
for each time series of the plurality of time series, selecting, by the one or more processors, the time series for inclusion in a reduced set of time series based on the newest timestamp associated with the time series and on the time selector, the reduced set of time series having a lower number of time series than a number of time series in the plurality of time series, wherein the time series is included in the reduced set of time series when the newest timestamp of the time series is greater than or equal to a lowest time in the time period of the time selector;
obtaining the reduced set of time series based on the selected time series; and
performing, by the one or more processors, the query on the reduced set of time series.
2. The method as recited in claim 1 , wherein the time-series metadata that includes the newest timestamp is stored in a metadata catalog, wherein selecting each of the plurality of time series comprises:
accessing the metadata catalog to obtain the newest timestamp in the metadata catalog.
3. The method as recited in claim 2 , further comprising:
periodically analyzing the metadata catalog to deter mine time series with the newest timestamp that is outside a predetermined time period; and
marking the determined time series as not current in the metadata catalog.
4. The method as recited in claim 3 , further comprising:
receiving a new query for a time interval within the predetermined time period; and
discarding from analysis of the new query the time series marked as not current.
5. The method as recited in claim 3 , further comprising:
moving the determined time series to an archival metadata catalog.
6. The method as recited in claim 2 , wherein the metadata catalog utilizes a hierarchical caching scheme with frequently accessed time series cached for quick access.
7. The method as recited in claim 2 , further comprising:
maintaining a time level indexing for the metadata catalog, the time level indexing being for a granularity of a predefined time period.
8. The method as recited in claim 1 , further comprising:
for each time series, maintaining in the time-series metadata a value of an oldest timestamp in the time series, the oldest timestamp being the timestamp stored first in the time series.
9. A system comprising:
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations comprising:
providing a plurality of time series, each time series comprising a plurality of values of a metric and a corresponding timestamp for each value of the metric, the plurality of values being indexed in order of the timestamps;
for each time series, maintaining time-series metadata stored separately from computer storage where the time series is stored, the time-series metadata comprising a value of a newest timestamp corresponding to a newest metric value stored in the time series, the newest timestamp being the timestamp stored last in the time series;
receiving a query, the query comprising a time selector for selecting entries from the plurality of time series within a time period;
for each time series of the plurality of time series, selecting, by the one or more processors, the time series for inclusion in a reduced set of time series based on the newest timestamp associated with the time series and on the time selector, the reduced set of time series having a lower number of time series than a number of time series in the plurality of time series, wherein the time series is included in the reduced set of time series when the newest timestamp of the time series is greater than or equal to a lowest time in the time period of the time selector;
obtaining the reduced set of time series based on the selected time series; and
performing the query on the reduced set of time series.
10. The system as recited in claim 9 , wherein the time-series metadata that includes the newest timestamp is stored in a metadata catalog, wherein selecting each of the plurality of time series comprises:
accessing the metadata catalog to obtain the newest timestamp in the metadata catalog.
11. The system as recited in claim 10 , wherein the instructions further cause the one or more computer processors to perform operations comprising:
periodically analyzing the metadata catalog to deter mine time series with the newest timestamp that is outside a predetermined time period; and
marking the determined time series as not current in the metadata catalog.
12. The system as recited in claim 11 , wherein the instructions further cause the one or more computer processors to perform operations comprising:
receiving a new query for a time interval within the predetermined time period; and
discarding from analysis of the new query the time series marked as not current.
13. The system as recited in claim 11 , wherein the instructions further cause the one or more computer processors to perform operations comprising:
moving the determined time series to an archival metadata catalog.
14. The system as recited in claim 10 , wherein the metadata catalog utilizes a hierarchical caching scheme with frequently accessed time series cached for quick access.
15. The system as recited in claim 10 , wherein the instructions further cause the one or more computer processors to perform operations comprising:
maintaining a time level indexing for the metadata catalog, the time level indexing being for a granularity of a predefined time period.
16. A tangible machine-readable storage medium including instructions that, when executed by a machine; cause the machine to perform operations comprising:
providing a plurality of time series, each time series comprising a plurality of values of a metric and a corresponding timestamp for each value of the metric, the plurality of values being indexed in order of the timestamps;
for each time series, maintaining time-series metadata stored separately from computer storage where the time series is stored, the time-series metadata comprising a value of a newest timestamp corresponding to a newest metric value stored in the time series, the newest timestamp being the timestamp stored last in the time series;
receiving a query, the query comprising a time selector for selecting entries from the plurality of time series within a time period;
for each time series of the plurality of time series, selecting the time series for inclusion in a reduced set of time series based on the newest timestamp associated with the time series and on the time selector, the reduced set of time series having a lower number of time series than a number of time series in the plurality of time series, wherein the time series is included in the reduced set of time series when the newest timestamp of the time series is greater than or equal to a lowest time in the time period of the time selector;
obtaining the reduced set of time series based on the selected time series; and
performing the query on the reduced set of time series.
17. The tangible machine-readable storage medium as recited in claim 16 , wherein the time-series metadata that includes the newest timestamp is stored in a metadata catalog, wherein selecting each of the plurality of time series comprises:
accessing the metadata catalog to obtain the newest timestamp in the metadata catalog.
18. The tangible machine-readable storage medium as recited in claim 17 , wherein the machine further performs operations comprising:
periodically analyzing the metadata catalog to determine time series with the newest timestamp that is outside a predetermined time period; and
marking the determined time series as not current in the metadata catalog.
19. The tangible machine-readable storage medium as recited in claim 18 , wherein the machine further performs operations comprising:
receiving a new query for a time interval within the predetermined time period; and
discarding from analysis of the new query the time series marked as not current.
20. The tangible machine-readable storage medium as recited in claim 18 , wherein the machine further performs operations comprising:
moving the determined time series to an archival metadata catalog.