Logs to metrics synthesis
A technique for logs to metrics synthesis is disclosed. A log message is received. It is determined that the received log message should be translated into a metrics data point. In response to determining that the received log message should be translated into a metrics data point, the metrics data point is generated using the received log message, the generated metrics data point comprising a timestamp, a metric name, a metric value, and a set of metadata key-value pairs. A time series in which to insert the metrics data point generated using the received log message is identified. The generated metrics data point is inserted into the identified time series.
1. A system, comprising:
a processor configured to:
ingest a log message comprising a string;
in response to determining that the log message matches a scope of a logs-to-metrics translation rule:
determine that the log message should be translated into a metrics data point; and
extract, using one or more parse expressions, one or more fields from the log message;
in response to determining that the log message should be translated into a metrics data point, translate the log message comprising the string into the metrics data point at least in part by generating the metrics data point using one or more values extracted from the log message, the generated metrics data point including a timestamp, a metric name, a metric value, and a set of metadata key-value pairs;
identify a time series in which to insert the metrics data point generated using the one or more values extracted from the log message, wherein the identifying comprises deriving a time series identifier based at least in part on at least a portion of the set of metadata key-value pairs included in the metrics data point generated using the log message;
insert the generated metrics data point into the identified time series; and
subsequent to inserting the generated metrics data point into the identified time series, receive a query for one or more time series; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 , wherein the processor is configured to generate the metrics data point at least in part by mapping a portion of the fields extracted from the log message to the metric name.
3. The system of claim 1 , wherein the processor is configured to generate the metrics data point at least in part by mapping a portion of the fields extracted from the log message to one or more dimensions associated with the set of metadata key-value pairs.
4. The system of claim 1 , wherein the processor is further configured to update a metadata catalog.
5. The system of claim 1 , wherein the processor is configured to identify the time series based at least in part on at least a portion of the set of metadata key-value pairs associated with the generated metrics data point.
6. The system of claim 1 , wherein the processor is further configured to determine whether to drop the log message used to generate the metrics data point or to keep the log message used to generate the metrics data point.
7. The system of claim 6 , wherein in response to determining that the log message used to generate the metrics data point should be kept, the processor is further configured to persist the log message.
8. The system of claim 7 , wherein the processor is further configured to associate an identifier of the log message with the metrics data point generated using the one or more values extracted from the log message.
9. A method, comprising:
ingesting a log message comprising a string;
in response to determining that the log message matches a scope of a logs-to-metrics translation rule:
determining that the log message should be translated into a metrics data point; and
extracting, using one or more parse expressions, one or more fields from the log message;
in response to determining that the log message should be translated into a metrics data point, translating the log message comprising the string into the metrics data point at least in part by generating the metrics data point using one or more values extracted from the log message, the generated metrics data point including a timestamp, a metric name, a metric value, and a set of metadata key-value pairs;
identifying a time series in which to insert the metrics data point generated using the one or more values extracted from the log message, wherein the identifying comprises deriving a time series identifier based at least in part on at least a portion of the set of metadata key-value pairs included in the metrics data point generated using the log message;
inserting the generated metrics data point into the identified time series; and
subsequent to inserting the generated metrics data point into the identified time series, receiving a query for one or more time series.
10. The method of claim 9 , wherein generating the metrics data point comprises mapping a portion of the fields extracted from the log message to the metric name.
11. The method of claim 9 , wherein generating the metrics data point comprises mapping a portion of the fields extracted from the log message to one or more dimensions associated with the set of metadata key-value pairs.
12. The method of claim 9 , further comprising updating a metadata catalog.
13. The method of claim 9 , wherein the time series is identified based at least in part on at least a portion of the set of metadata key-value pairs associated with the generated metrics data point.
14. The method of claim 9 , further comprising determining whether to drop the log message used to generate the metrics data point or to keep the log message used to generate the metrics data point.
15. The method of claim 14 , further comprising persisting the log message in response to determining that the log message used to generate the metrics data point should be kept.
16. The method of claim 15 , further comprising associating an identifier of the log message with the metrics data point generated using the one or more values extracted from the log message.
17. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
ingesting a log message comprising a string;
in response to determining that the log message matches a scope of a logs-to-metrics translation rule:
determining that the log message should be translated into a metrics data point; and
extracting, using one or more parse expressions, one or more fields from the log message;
in response to determining that the log message should be translated into a metrics data point, translating the log message comprising the string into the metrics data point at least in part by generating the metrics data point using one or more values extracted from the log message, the generated metrics data point including a timestamp, a metric name, a metric value, and a set of metadata key-value pairs;
identifying a time series in which to insert the metrics data point generated using the one or more values extracted from the log message, wherein the identifying comprises deriving a time series identifier based at least in part on at least a portion of the set of metadata key-value pairs included in the metrics data point generated using the log message;
inserting the generated metrics data point into the identified time series; and
subsequent to inserting the generated metrics data point into the identified time series, receiving a query for one or more time series.