Identifying configuration parameters for a query using a metadata catalog
Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system parses the query and uses a metadata catalog to dynamically identify configuration parameters of datasets and/or rules associated with the query. The identified configuration parameters are communicated to a query processing component of the data intake and query system for use in executing the query.
1. A method, comprising:
receiving a query, the query identifying a set of data and a manner of processing the set of data;
identifying one or more datasets associated with the query;
obtaining configuration parameters associated with at least one dataset of the one or more datasets from a metadata catalog; and
communicating the configuration parameters associated with the at least one dataset to a query processing component of a data intake and query system.
2. The method of claim 1 , wherein the query is received via a gateway application programming interface.
3. The method of claim 1 , further comprising receiving a dataset association record identifier associated with the query, wherein said identifying the one or more datasets is based at least in part on the dataset association record identifier.
4. The method of claim 1 , wherein the query includes a dataset identifier, and wherein said identifying the one or more datasets is based at least in part on the dataset identifier.
5. The method of claim 1 , wherein the query includes a command to retrieve a dataset of the one or more datasets.
6. The method of claim 1 , wherein the query includes a dataset identifier, the method further comprising receiving a dataset association record identifier associated with the query, wherein said identifying the one or more datasets is based at least in part on the dataset identifier and the dataset association record identifier.
7. The method of claim 1 , wherein the query includes a first dataset identifier, the method further comprising:
receiving a dataset association record identifier associated with the query; and
determining a second dataset identifier based at least in part on the first dataset identifier and the dataset association record identifier,
wherein said identifying the one or more datasets is based at least in part on the second dataset identifier.
8. The method of claim 1 , wherein said identifying the one or more datasets comprises:
identifying a first dataset of the one or more datasets based at least in part on a dataset identifier associated with the query;
parsing the first dataset to identify one or more additional datasets referenced by the first dataset; and
recursively parsing the one or more additional datasets to identify the one or more datasets associated with the query.
9. The method of claim 1 , wherein the one or more datasets are directly referenced by the query or referenced by another dataset that is directly or indirectly referenced by the query.
10. The method of claim 1 , wherein the one or more datasets are a subset of a plurality of datasets of a dataset association record of the metadata catalog.
11. The method of claim 1 , wherein the one or more datasets are a subset of a plurality of datasets of a group of dataset association records of the metadata catalog.
12. The method of claim 1 , wherein the one or more datasets are a subset of a plurality of datasets of the metadata catalog.
13. The method of claim 1 , wherein the at least one dataset includes at least a portion of the set of data.
14. The method of claim 1 , wherein the at least one dataset references at least a portion of the set of data.
15. The method of claim 1 , wherein the at least one dataset does not reference another dataset.
16. The method of claim 1 , wherein the at least one dataset is identified for use during execution of the query.
17. The method of claim 1 , further comprising identifying the at least one dataset, wherein identifying the at least one dataset comprises:
identifying a first dataset of the one or more datasets based at least in part on a dataset identifier associated with the query;
parsing the first dataset to identify one or more additional datasets referenced by the first dataset; and
recursively parsing the one or more additional datasets to identify the at least one dataset.
18. The method of claim 1 , further comprising identifying the at least one dataset, wherein identifying the at least one dataset comprises:
identifying a first dataset of the one or more datasets based at least in part on a dataset identifier associated with the query;
parsing the first dataset to identify one or more additional datasets referenced by the first dataset; and
recursively parsing the one or more additional datasets to identify the at least one dataset, wherein the at least one dataset includes at least a portion of the set of data.
19. The method of claim 1 , further comprising identifying the at least one dataset, wherein identifying the at least one dataset comprises:
identifying a first dataset of the one or more datasets based at least in part on a dataset identifier associated with the query;
parsing the first dataset to identify one or more additional datasets referenced by the first dataset; and
recursively parsing the one or more additional datasets to identify the at least one dataset, wherein the at least one dataset does not reference another dataset.
20. The method of claim 1 , further comprising:
identifying one or more rules associated with the at least one dataset; and
obtaining configuration parameters associated with the one or more rules, wherein the configuration parameters associated with at least one dataset comprise the configuration parameters associated with the one or more rules.
21. The method of claim 1 , further comprising:
identifying one or more rules for processing data of the at least one dataset; and
obtaining configuration parameters associated with the one or more rules, wherein the configuration parameters associated with at least one dataset comprise the configuration parameters associated with the one or more rules.
22. The method of claim 1 , wherein the query processing component processes and executes the query based at least in part on the configuration parameters.
23. The method of claim 1 , wherein the query processing component uses the configuration parameters to access the at least one dataset.
24. The method of claim 1 , wherein the query processing component uses the configuration parameters to obtain data of the at least one dataset.
25. The method of claim 1 , wherein the query processing component uses the configuration parameters to parse data of the at least one dataset.
26. The method of claim 1 , further comprising:
determining that the query processing component is not responsive; and
communicating the configuration parameters to another query processing component.
27. The method of claim 1 , further comprising communicating the query to the query processing component with the configuration parameters.
28. The method of claim 1 , further comprising:
modifying the query based at least in part on the one or more datasets; and
communicating a modified query to the query processing component based at least in part on said modifying the query.
29. A computing system of a data intake and query system, the computing system comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive a query, the query identifying a set of data and a manner of processing the set of data;
identify one or more datasets associated with the query;
obtain configuration parameters associated with at least one dataset of the one or more datasets from a metadata catalog; and
communicate the configuration parameters associated with the at least one dataset to a query processing component of the data intake and query system.
30. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive a query, the query identifying a set of data and a manner of processing the set of data;
identify one or more datasets associated with the query;
obtain configuration parameters associated with at least one dataset of the one or more datasets from a metadata catalog; and
communicate the configuration parameters associated with the at least one dataset to a query processing component of the data intake and query system.