Conversion of cloud computing platform data for ingestion by data intake and query system
In accordance with various embodiments of the present disclosure, a query for information related to machine data generated by one or more machine data sources of a cloud computing platform (CCP) is sent by a client computing device and to a cloud computing monitoring component of the CCP, where the query is formed using native query language of the CCP. As a result, the client computing device via a connector receives a first data object that is formatted in accordance with a first format associated with the CCP. The client computing device via the connector may then convert the first data object to one or more second data objects formatted in accordance with a second format that allows for enhanced ingestion by a data intake and query system.
1. A computer-implemented method, comprising:
sending, by a client computing device and to a cloud computing monitoring component of a cloud computing platform (CCP), a query for information related to machine data generated by one or more machine data sources of the CCP, wherein the query is formed using native query language of the CCP, the cloud computing monitoring component monitoring performances of one or more components operating in the CCP;
receiving, by the client computing device via a connector, a first data object formatted in accordance with a first format associated with the CCP;
converting, by the client computing device via the connector, the first data object to one or more second data objects formatted in accordance with a second format that allows for enhanced ingestion by a data intake and query system (DIQS),
wherein the first data object includes a plurality of metrics, and wherein converting the first data object to the one or more second data objects includes converting the first data object into a plurality of second data objects such that each of the plurality of second data objects includes a single metric from amongst the plurality of metrics included in the first data object,
wherein each of the plurality of second data objects includes same information including information indicating a common source not included in the first data object: and
providing, by the client computing device and to the DIQS, the plurality of second data objects.
2. The computer-implemented method of claim 1 , wherein the query is generated based on a query string obtained from the CCP.
3. The computer-implemented method of claim 1 , wherein sending of the query is by initiating a script that implements the connector causing the query to be automatically sent during runtime of the script.
4. The computer-implemented method of claim 1 , wherein the connector is implemented by the client computing device and the client computing device is external to the CCP.
5. The computer-implemented method of claim 4 , wherein the client computing device is associated with the DIQS.
6. The computer-implemented method of claim 1 , wherein the plurality of metrics of the first data object were produced by the cloud computing monitoring component processing the machine data generated by the one or more machine data sources.
7. The computer-implemented method of claim 6 , wherein the plurality of metrics was produced by aggregating at least a portion of the machine data.
8. The computer-implemented method of claim 1 , wherein the first data object includes information related to log data generated by the one or more machine data sources.
9. The computer-implemented method of claim 1 , wherein converting the first data object to the plurality second data objects includes tagging each of the plurality of second data objects with metadata that includes information that is parsed during a search for content of the one or more second data objects when the content has been stored in a metrics store.
10. The computer-implemented method of claim 1 , wherein each of the first data object and each of the plurality of second data objects is a JavaScript Object Notation (JSON) object.
11. The computer-implemented method of claim 1 , wherein each of the plurality of second data objects includes a key value and a measured value taken from a machine data source.
12. The computer-implemented method of claim 1 , wherein providing the plurality of second data objects include providing the plurality of second data objects each with an HTTP event collector (HEC) token to ensure that the plurality of second data objects will be accepted by the DIQS.
13. The computer-implemented method of claim 1 , wherein the plurality of second data objects are provided to one or more indexers of the DIQS.
14. The computer-implemented method of claim 1 , wherein the DIQS is configured to ingest unstructured machine data.
15. The computer-implemented method of claim 14 , wherein the DIQS is configured to use a data model that includes a late binding schema of one or more extraction rules applied at search time.
16. The computer-implemented method of claim 1 , wherein the DIQS comprises a time-series data store configured to receive machine data processed into time stamped events of unstructured data that are accessible via search queries.
17. The computer-implemented method of claim 1 , wherein the DIQS is configured to store a plurality of metrics in an index of a metrics store, wherein each metric of the plurality of metrics includes at least one key value and a measured value obtained from a machine data resource.
18. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
send, by a client computing device and to a cloud computing monitoring component of a cloud computing platform (CCP), a query for information related to machine data generated by one or more machine data sources of the CCP, wherein the query is formed using native query language of the CCP, the cloud computing monitoring component monitoring performance of one or more components operating in the CCP;
receive, by the client computing device via a connector, a first data object formatted in accordance with a first format associated with the CCP;
convert, by the client computing device via the connector, the first data object to one or more second data objects formatted in accordance with a second format that allows for enhanced ingestion by a data intake and query system (DIQS);
wherein the first data object includes a plurality of metrics, and wherein converting the first data object to the one or more second data objects includes converting the first data object into a plurality of second data objects such that each of the plurality of second data objects includes a single metric from amongst the plurality of metrics included in the first data object,
wherein each of the plurality of second data objects includes same information including information indicating a common source not included in the first data object: and
provide, by the client computing device and to the DIQS, the plurality of second data objects.
19. The one or more non-transitory computer-readable storage media of claim 18 , wherein converting the first data object to the plurality of second data objects includes tagging each of the plurality second data objects with metadata that includes information that is parsed during a search for content of the one or more second data objects when the content has been stored in a metric store.
20. The one or more non-transitory computer-readable storage media of claim 18 , wherein the instructions when executed by the one or more processors further causes the one or more processors to perform the step of providing the plurality of second data objects each with an HTTP event collector (HEC) token to ensure that the plurality of second data objects will be accepted by the DIQS.
21. A computing device, comprising:
one or more processors; and
memory containing instructions that, when executed by the one or more processors, cause the computing device to:
send, to a cloud computing monitoring component of a cloud computing platform (CCP), a query for information related to machine data generated by one or more machine data sources of the CCP, wherein the query is formed using native query language of the CCP, the cloud computing monitoring component monitoring performance of one or more components operating in the CCP;
receive a first data formatted in accordance with a first format associated with the CCP;
convert the first data object to one or more second data objects formatted in accordance with a second format that allows for enhanced ingestion by a data intake and query system (DIQS);
wherein the first data object includes a plurality of metrics, and wherein converting the first data object to the one or more second data objects includes converting the first data object into a plurality of second data objects such that each of the plurality of second data objects includes a single metric from amongst the plurality of metrics included in the first data object,
wherein each of the plurality of second data objects includes same information including information indicating a common source not included in the first data object: and
provide the plurality of second data objects to the DIQS.