AUTOMATIC TRANSFORMATION OF TIME SERIES DATA AT INGESTION
In a computer-implemented method for automatic transformation of time series data at ingestion, time series data comprising data points is received at at least one ingestion node of a time series data monitoring system, wherein the data points have an input observability format. At the at least one ingestion node, the data points the data points are transformed from the input observability format to an output observability format according to configuration rules of the time series data monitoring system. The data points having the output observability format are forwarded from the at least one ingestion node to a persistent storage device.
1 . A computer-implemented method for automatic transformation of time series data at ingestion, the method comprising:
receiving time series data comprising data points at at least one ingestion node of a time series data monitoring system, wherein the data points have an input observability format;
transforming, at the at least one ingestion node, the data points from the input observability format to an output observability format according to configuration rules of the time series data monitoring system; and
forwarding, from the at least one ingestion node, the data points having the output observability format to a persistent storage device.
2 . The method of claim 1 , further comprising:
forwarding, from the at least one ingestion node, the data points having the input observability format to the persistent storage device.
3 . The method of claim 1 , further comprising:
deleting the data points comprising the input observability format subsequent transformation to the output observability format.
4 . The method of claim 1 , further comprising:
receiving subsets of data points having the output observability format from a plurality of ingestion nodes at an intermediate aggregation node between the plurality of ingestion nodes and the persistent storage device; and
aggregating the subsets of data points having the output observability format from the plurality of ingestion nodes into aggregated data points having the output observability format.
5 . The method of claim 4 , further comprising:
forwarding, from the intermediate aggregation node, the aggregated data points having the output observability format to the persistent storage device.
6 . The method of claim 1 , further comprising:
receiving, at the at least one ingestion node, a stray data point of the time series data, the stray data point received subsequent the forwarding of the data points having the output observability format to the persistent storage device, wherein the stray data point has the input observability format;
transforming, at the at least one ingestion node, the stray data point from the input observability format to the output observability format according to the configuration rules of the time series data monitoring system; and
forwarding, from the at least one ingestion node, the stray data point having the output observability format to the persistent storage device.
7 . The method of claim 6 , further comprising:
responsive to receiving a query request associated with the data points having the output observability format and the stray data point having the output observability format, aggregating the data points having the output observability format and the stray data point having the output observability format into a complete set of aggregated data points having the output observability format; and
returning a result to the query request using the complete set of aggregated data points.
8 . The method of claim 1 , wherein the configuration rules of the time series data monitoring system define operations for the transforming the data points from the input observability format to the output observability format.
9 . The method of claim 1 , wherein the input observability format is one of a metric, a counter, a histogram, and a span.
10 . The method of claim 1 , wherein the output observability format is one of a counter and a histogram.
11 . A non-transitory computer readable storage medium having computer readable program code stored thereon for causing a computer system to perform a method for automatic transformation of time series data at ingestion, the method comprising:
receiving time series data comprising data points at at least one ingestion node of a time series data monitoring system, wherein the data points have an input observability format;
transforming, at the at least one ingestion node, the data points from the input observability format to an output observability format according to configuration rules of the time series data monitoring system; and
forwarding, from the at least one ingestion node, the data points having the output observability format to a persistent storage device.
12 . The non-transitory computer readable storage medium of claim 11 , the method further comprising:
forwarding, from the at least one ingestion node, the data points having the input observability format to the persistent storage device.
13 . The non-transitory computer readable storage medium of claim 11 , the method further comprising:
deleting the data points comprising the input observability format subsequent transformation to the output observability format.
14 . The non-transitory computer readable storage medium of claim 11 , the method further comprising:
receiving subsets of data points having the output observability format from a plurality of ingestion nodes at an intermediate aggregation node between the plurality of ingestion nodes and the persistent storage device; and
aggregating the subsets of data points having the output observability format from the plurality of ingestion nodes into aggregated data points having the output observability format.
15 . The non-transitory computer readable storage medium of claim 14 , the method further comprising:
forwarding, from the intermediate aggregation node, the aggregated data points having the output observability format to the persistent storage device.
16 . The non-transitory computer readable storage medium of claim 11 , the method further comprising:
receiving, at the at least one ingestion node, a stray data point of the time series data, the stray data point received subsequent the forwarding of the data points having the output observability format to the persistent storage device, wherein the stray data point has the input observability format;
transforming, at the at least one ingestion node, the stray data point from the input observability format to the output observability format according to the configuration rules of the time series data monitoring system; and
forwarding, from the at least one ingestion node, the stray data point having the output observability format to the persistent storage device.
17 . The non-transitory computer readable storage medium of claim 16 , the method further comprising:
responsive to receiving a query request associated with the data points having the output observability format and the stray data point having the output observability format, aggregating the data points having the output observability format and the stray data point having the output observability format into a complete set of aggregated data points having the output observability format; and
returning a result to the query request using the complete set of aggregated data points.
18 . The non-transitory computer readable storage medium of claim 11 , wherein the configuration rules of the time series data monitoring system define operations for the transforming the data points from the input observability format to the output observability format.
19 . The non-transitory computer readable storage medium of claim 11 , wherein the input observability format is one of a metric, a counter, a histogram, and a span and the output observability format is one of a counter and a histogram.
20 . A time series data monitoring system for automatic transformation of time series data at ingestion, the time series data monitoring system comprising:
a persistent storage device;
a plurality of ingestion nodes, each node of the plurality of ingestion nodes comprising a data storage unit and a processor communicatively coupled with the data storage unit, wherein an ingestion node of the plurality of ingestion nodes is configured to:
receive time series data comprising data points, wherein the data points have an input observability format;
transform the data points from the input observability format to an output observability format according to configuration rules of the time series data monitoring system; and
forwarding the data points having the output observability format to the persistent storage device.