Generating a subquery for an external data system using a configuration file
Systems and methods are disclosed for receiving, at a data intake and query system, a query that includes an indication to process data managed by a third-party data storage and processing system that supports a different query language than the data intake and query system. The data intake and query system identifies a third-party data storage and processing system that manages the data to be processed and generates a subquery for execution by the third-party data storage and processing system, generates instructions for one or more worker nodes to receive and process results of the subquery from the third-party data storage and processing system, and instructs the worker nodes to provide results of the processing to the data intake and query system.
1. A method, comprising:
receiving, at a data intake and query system, a query identifying a set of data to be processed and a manner of processing the set of data, wherein at least a portion of the query is written in a first query language interpretable by the data intake and query system, and wherein a subquery included in the query is written in a second query language interpretable by an external data system;
converting the subquery from the second query language to the first query language to obtain a translated subquery;
performing an optimization action on the translated subquery to obtain a processed translated subquery;
converting the processed translated subquery from the first query language to the second query language to obtain a modified subquery interpretable by the external data system;
defining, by the data intake and query system, a query processing scheme for obtaining and processing the set of data based on the query and the subquery, wherein defining the query processing scheme comprises generating instructions for one or more processors to receive and process results of the modified subquery to obtain processed results and to provide the processed results to the data intake and query system; and
executing the query based on the query processing scheme.
2. The method of claim 1 , wherein generating the instructions for the one or more processors comprises generating the instructions for one or more worker nodes to receive and process the results of the modified subquery.
3. The method of claim 1 , wherein performing the optimization action on the translated subquery comprises:
replacing the subquery in the query with the translated subquery to obtain a second query; and
performing the optimization action on the second query.
4. The method of claim 1 , wherein performing the optimization action on the translated subquery comprises:
replacing the subquery in the query with the translated subquery to obtain a second query; and
substituting one or more commands in the second query with a substitute command.
5. The method of claim 1 , wherein performing the optimization action on the translated subquery comprises substituting one or more commands in the translated subquery with a substitute command.
6. The method of claim 1 , wherein defining the query processing scheme further comprises:
determining a data ingest estimate for the subquery,
determining a partition size based on resources allocated to the query, one or more search parameters of the subquery, and/or a number of fields used to process events from the external data system, and
determining a number of partitions based on the partition size and the data ingest estimate, wherein the instructions for the one or more processors are generated based on the determined number of partitions.
7. The method of claim 1 , wherein defining the query processing scheme further comprises associating a first search identifier with the external data system; and wherein executing the query comprises:
receiving, by the one or more processors, the results of the subquery, wherein the results of the subquery include a second search identifier assigned to the results of the subquery by the external data system;
mapping the first search identifier to the second search identifier; and
processing the results of the subquery based on said mapping.
8. The method of claim 1 , wherein generating instructions for the one or more processors comprises:
obtaining a data ingest estimate for the subquery; and
generating instructions for the one or more processors based on the data ingest estimate.
9. The method of claim 1 , wherein generating instructions for the one or more processors comprises:
determining a quantity of partitions to ingest the results of the subquery; and
generating instructions for the one or more processors based on the quantity of partitions.
10. The method of claim 1 , wherein defining the query processing scheme further comprises obtaining network access information for a system comprising at least one of the one or more processors, and wherein executing the query comprises communicating the network access information to the external data system.
11. The method of claim 1 , wherein the external data system is one of a plurality of external data systems configured to communicate with the data intake and query system, and wherein at least one external data system of the plurality of external data systems is configured to interpret a different query language than at least one other external data system of the plurality of external data systems.
12. The method of claim 1 , wherein converting the subquery from the second query language to the first query language to obtain the translated subquery comprises:
parsing a syntax of the subquery to determine a command included in the subquery for the external data system; and
substituting the command included in the subquery for the external data system with at least one command that is of the first query language interpretable by the data intake and query system to obtain the translated subquery, wherein the command included in the subquery for the external data system is of the second query language interpretable by the external data system.
13. The method of claim 1 , wherein converting the subquery from the second query language to the first query language to obtain the translated subquery comprises:
determining an identity of the second query language interpretable by the external data system;
identifying a mapping of the second query language interpretable by the external data system to the first query language interpretable by the data intake and query system based at least in part on the identity of the second query language interpretable by the external data system; and
converting the subquery into the translated subquery based at least in part on the mapping.
14. The method of claim 1 , wherein converting the subquery from the second query language to the first query language converting the second query language into a semantically equivalent query in the first query language.
15. The method of claim 1 , further comprising determining based on a keyword that the query includes the subquery for the external data system.
16. The method of claim 1 , further comprising determining the subquery based on a query parameter included in the query.
17. The method of claim 1 , further comprising determining the subquery based on a query parameter included in the query, wherein the query parameter is used as an index to a configuration file to identify the subquery from one or more queries included in the configuration file.
18. The method of claim 1 , wherein the external data system is a third-party data storage and processing system that differs from the data intake and query system.
19. A computing system of a data intake and query system, the computing system comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive, at the data intake and query system, a query identifying a set of data to be processed and a manner of processing the set of data, wherein at least a portion of the query is written in a first query language interpretable by the data intake and query system, and wherein a subquery included in the query is written in a second query language interpretable by an external data system;
convert the subquery from the second query language to the first query language to obtain a translated subquery;
perform an optimization action on the translated subquery to obtain a processed translated subquery;
convert the processed translated subquery from the first query language to the second query language to obtain a modified subquery interpretable by the external data system;
define, by the data intake and query system, a query processing scheme for obtaining and processing the set of data based on the query and the subquery, wherein defining the query processing scheme comprises generating instructions for one or more processors to receive and process results of the modified subquery to obtain processed results and to provide the processed results to the data intake and query system; and
execute the query based on the query processing scheme.
20. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive, at the data intake and query system, a query identifying a set of data to be processed and a manner of processing the set of data, wherein at least a portion of the query is written in a first query language interpretable by the data intake and query system, and wherein a subquery included in the query is written in a second query language interpretable by an external data system;
convert the subquery from the second query language to the first query language to obtain a translated subquery;
perform an optimization action on the translated subquery to obtain a processed translated subquery;
convert the processed translated subquery from the first query language to the second query language to obtain a modified subquery interpretable by the external data system;
define, by the data intake and query system, a query processing scheme for obtaining and processing the set of data based on the query and the subquery, wherein defining the query processing scheme comprises generating instructions for one or more processors to receive and process results of the modified subquery to obtain processed results and to provide the processed results to the data intake and query system; and
execute the query based on the query processing scheme.