Generating a subquery for an external data system using a configuration file
Systems and methods are disclosed for receiving, at a data intake and query system, a query that includes an indication to process data managed by a third-party data storage and processing system that supports a different query language than the data intake and query system. The data intake and query system identifies a third-party data storage and processing system that manages the data to be processed and generates a subquery for execution by the third-party data storage and processing system, generates instructions for one or more worker nodes to receive and process results of the subquery from the third-party data storage and processing system, and instructs the worker nodes to provide results of the processing to the data intake and query system.
1. A method, comprising:
receiving, at a query processing system, a first query, wherein a first query portion of the first query is written in a first query language interpretable by the query processing system, and wherein a second query portion of the first query is written in a second query language interpretable by an external data system that is separate from the query processing system;
translating the second query portion of the first query from the second query language to the first query language to obtain a first translated query portion;
performing a query refinement action on the first query based at least in part on the first translated query portion to obtain a second query;
translating a first portion of the second query from the first query language to the second query language to obtain a second translated query portion, wherein the second translated query portion is interpretable by the external data system; and
executing the second query by at least causing the second translated query portion to be executed at the external data system.
2. The method of claim 1 , wherein performing the query refinement action comprises modifying the first translated query portion to obtain the second query.
3. The method of claim 1 , wherein performing the query refinement action comprises omitting the first translated query portion to obtain the second query.
4. The method of claim 1 , wherein the query refinement action is determined based at least in part on the first query portion.
5. The method of claim 1 , wherein performing the query refinement action comprises replacing a command included in the first translated query portion with an alternative command to obtain the second query.
6. The method of claim 1 , wherein performing the query refinement action comprises modifying the first query portion to obtain the second query.
7. The method of claim 1 , wherein performing the query refinement action comprises omitting the first query portion to obtain the second query.
8. The method of claim 1 , wherein performing the query refinement action comprises modifying both the first query portion and the second query portion to obtain the second query.
9. The method of claim 1 , wherein translating the first portion of the second query from the first query language to the second query language comprises translating the first translated query portion of the first query to the second query language.
10. The method of claim 1 , wherein translating the first portion of the second query from the first query language to the second query language comprises translating a modified version of the first translated query portion of the first query to the second query language.
11. The method of claim 1 , further comprising translating a second portion of the second query from the first query language to a third query language to obtain a third translated query portion, wherein the third translated query portion is interpretable by a second external data system.
12. The method of claim 1 , further comprising generating instructions for one or more processors to execute the second query and to provide results of the second query to the query processing system.
13. The method of claim 1 , further comprising:
determining a data ingest estimate for the first portion of the second query,
determining a partition size based on resources allocated to the first query, one or more search parameters of the first portion of the second query, and/or a number of fields used to process events from the external data system, and
determining a number of partitions based on the partition size and the data ingest estimate.
14. The method of claim 13 , further comprising generating instructions for one or more processors to execute the second query based at least in part on the number of partitions.
15. The method of claim 1 , wherein translating the second query portion of the first query from the second query language to the first query language comprises accessing a mapping in a configuration file that maps a command in the second query language to a command in the first query language.
16. The method of claim 1 , wherein translating the second query portion of the first query from the second query language to the first query language to obtain the first translated query portion comprises using at least a part of the second query portion as an index value to access a configuration file that comprises mappings between commands of the second query language and the first query language.
17. The method of claim 1 , wherein the query refinement action comprises a semantic-based optimization, a runtime-based optimization, or an infrastructure-based optimization.
18. The method of claim 1 , wherein the external data system is a third-party query processing system that differs from the query processing system.
19. A computing system of a query processing system, the computing system comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive a first query, wherein a first query portion of the first query is written in a first query language interpretable by the query processing system, and wherein a second query portion of the first query is written in a second query language interpretable by an external data system that is separate from the query processing system;
translate the second query portion of the first query from the second query language to the first query language to obtain a first translated query portion;
perform a query refinement action on the first query based at least in part on the first translated query portion to obtain a second query;
translate a first portion of the second query from the first query language to the second query language to obtain a second translated query portion, wherein the second translated query portion is interpretable by the external data system; and
execute the second query by at least causing the second translated query portion to be executed at the external data system.
20. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system of a query processing system, cause the computing system to:
receive a first query, wherein a first query portion of the first query is written in a first query language interpretable by the query processing system, and wherein a second query portion of the first query is written in a second query language interpretable by an external data system that is separate from the query processing system;
translate the second query portion of the first query from the second query language to the first query language to obtain a first translated query portion;
perform a query refinement action on the first query based at least in part on the first translated query portion to obtain a second query;
translate a first portion of the second query from the first query language to the second query language to obtain a second translated query portion, wherein the second translated query portion is interpretable by the external data system; and
execute the second query by at least causing the second translated query portion to be executed at the external data system.