Data enrichment and augmentation
Data enrichment and augmentation is disclosed. Machine data comprising at least one of a log message and a metrics data point is received. The received machine data comprises an identifier of an instance of a virtual machine. Based at least in part on the identifier of the instance of the virtual machine, a query for tags associated with the instance of the virtual machine is performed. At least one key-value pair is generated based at least in part on tags received in response to the query performed based at least in part on the identifier of the instance of the virtual machine. The received machine data is augmented with the at least one key-value pair generated based at least in part on the tags received in response to the query based at least in part on the identifier of the instance of the virtual machine.
1. A system, comprising:
a processor configured to:
receive, from a virtual machine executing on a cloud service, machine data comprising a metrics data point generated by the virtual machine and an identifier of an instance of the virtual machine;
using the identifier of the instance of the virtual machine, query the cloud service for tags associated with the instance of the virtual machine, the tags comprising information about the cloud service where the virtual machine is executing;
generate at least one key-value pair based on the tags received from the cloud service;
augment the machine data by adding, to the metrics data point, the generated at least one key-value pair, wherein the adding of the generated at least one key-value pair to the metrics data point is performed by joining the machine data with the identifier of the instance of the virtual machine and the generated at least one key-value pair; and
insert the augmented machine data into a time series for the instance of the virtual machine; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 wherein the instance of the virtual machine is configured to execute a collector, and wherein at least a portion of the received machine data is received from the collector executing on the instance of the virtual machine.
3. The system of claim 2 wherein the collector is configured to locally perform a lookup of the identifier of the instance of the virtual machine on which the collector is executed.
4. The system of claim 1 wherein querying the cloud service comprises querying an application programming interface (API) associated with the cloud service.
5. The system of claim 1 wherein querying the cloud service comprises providing, to the cloud service, a set of credentials.
6. The system of claim 1 wherein the received tags comprise a first tag specifying a geographical region in which the instance of the virtual machine is running.
7. The system of claim 1 wherein the received tags comprise a second tag specifying an instance type of the instance of the virtual machine, wherein the instance type comprises one of a graphics processing unit (GPU), a field programmable gate array (FPGA), and a central processing unit (CPU).
8. The system of claim 1 wherein the received tags comprise a third tag indicating a deployment status of the instance of the virtual machine, wherein the deployment status comprises at least one of production and staging.
9. A method, comprising:
receiving, from a virtual machine executing on a cloud service, machine data comprising a metrics data point generated by the virtual machine and an identifier of an instance of the virtual machine;
using the identifier of the instance of the virtual machine, querying the cloud service for tags associated with the instance of the virtual machine, the tags comprising information about the cloud service where the virtual machine is executing;
generating at least one key-value pair based on the tags received from the cloud service;
augmenting the machine data by adding, to the metrics data point, the generated at least one key-value pair, wherein the adding of the generated at least one key-value pair to the metrics data point is performed by joining the machine data with the identifier of the instance of the virtual machine and the generated at least one key-value pair; and
insert the augmented machine data into a time series for the instance of the virtual machine.
10. The method of claim 9 wherein the instance of the virtual machine is configured to execute a collector, and wherein at least a portion of the received machine data is received from the collector executing on the instance of the virtual machine.
11. The method of claim 10 wherein the collector is configured to locally perform a lookup of the identifier of the instance of the virtual machine on which the collector is executed.
12. The method of claim 9 wherein querying the cloud service comprises querying an application programming interface (APO) associated with the cloud service.
13. The method of claim 9 wherein querying the cloud service comprises providing, to the cloud service, a set of credentials.
14. The method of claim 9 wherein the received tags comprise a first tag specifying a geographical region in which the instance is running.
15. The method of claim 9 wherein the received tags comprise a second tag specifying an instance type of the instance of the virtual machine, wherein the instance type comprises one of a graphics processing unit (GPU), a field programmable gate array (FPGA), and a central processing unit (CPU).
16. The method of claim 9 wherein the received tags comprise a third tag indicating a deployment status of the instance of the virtual machine, wherein the deployment status comprises at least one of production and staging.
17. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving, from a virtual machine executing on a cloud service, machine data comprising a metrics data point generated by the virtual machine and an identifier of an instance of the virtual machine;
using the identifier of the instance of the virtual machine, querying the cloud service for tags associated with the instance of the virtual machine, the tags comprising information about the cloud service where the virtual machine is executing;
generating at least one key-value pair based on the tags received from the cloud service;
augmenting the machine data by adding, to the metrics data point, the generated at least one key-value pair, wherein the adding of the generated at least one key-value pair to the metrics data point is performed by joining the machine data with the identifier of the instance of the virtual machine and the generated at least one key-value pair; and
insert the augmented machine data into a time series for the instance of the virtual machine.
18. The system of claim 1 wherein receiving the machine data comprises collecting the at least one of a log message and the metrics data point from the virtual machine.
19. The system of claim 1 wherein the key-value pair includes a name of the tag received from the cloud service and a value of the tag received from the cloud service.
20. The system of claim 1 wherein the received tags comprise a fourth tag specifying an identifier of a host where the virtual machine is executing.