Reading query results from an external data system
A computing device can receive a query in a first query language that identifies a set of data to be processed and determine that at least a portion of the set of data resides in an external data system that uses a different query language. The query system can translate the query in the first query language in to a second query language for the external data system. To retrieve results of the translated query, the computing device may determine a quantity of results of generated by the translated query and generate one or more results readers to read the results in parallel. The computing device may further process the results and provide them to a user.
1 . A method, comprising:
receiving, at a computing device of a first query system, a first query in a first query language, the first query identifying a set of data and a manner of processing the set of data, wherein the first query includes a set of commands;
processing the first query to determine the set of data includes a dataset residing in a shared storage system and accessible via a second query system using a second query language that is different from the first query language;
translating the first query into a second query in the second query language by:
identifying a first query command in the first query language that has a corresponding query command in the second query language,
translating the first query command in the first query language to the second query language based on the first query command,
identifying a second query command in the first query language for which no corresponding query command exists in the second query language, and
generating a new query command in the second query language that performs functions similar to the second query command in the first query language; and
communicating the second query to a search coordinator of the second query system, wherein the search coordinator executes the second query on the dataset and generates search results.
2 . The method of claim 1 further comprising:
determining a location of the search results;
determining a quantity of the search results;
generating a plurality of results readers based on the determined quantity of the search results;
concurrently retrieving the search results from the determined location using the generated plurality of results readers;
processing the search results to determine a set of search results; and
communicating the set of search results to a second computing device associated with a user.
3 . The method of claim 2 , wherein processing the search results comprises changing a format of the search results to a second format to generate the set of search results, wherein communicating the set of search results comprises communicating the set of search results in the second format.
4 . The method of claim 2 , wherein processing the search results comprises combining the search results with additional search results received from a third query system to determine the set of search results.
5 . The method of claim 1 , wherein the first query system and the search coordinator of the second query system are instantiated in a shared computing resource environment of a cloud service provider and are associated with a first cloud service account of the cloud service provider, and
wherein the dataset stored in a first location comprises a portion of the shared storage system of the cloud service provider and is associated with a second cloud service account of the cloud service provider.
6 . The method of claim 1 , wherein the first query system and the search coordinator of the second query system are instantiated in a shared computing resource environment of a cloud service provider and are associated with a first cloud service account of the cloud service provider,
wherein the dataset stored in a first location comprises a first storage system of the cloud service provider and is associated with a second cloud service account of the cloud service provider.
7 . The method of claim 1 , wherein generating the new query command in the second query language comprises generating one or more query statements, query commands, system query parameters, lines of a query, or a combination thereof, in the second query language.
8 . The method of claim 1 , wherein generating the new query command in the second query language comprises using one or more user query parameters to determine which and how many query commands or system query parameters to generate in the second query language.
9 . The method of claim 1 , wherein the second query command in the first query language comprises a time-based query command related to organizing, sorting, and/or selecting events based on an associated time or timestamp.
10 . Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing device of a first query system, cause the computing device to:
receive, at the computing device of the first query system, a first query in a first query language, the first query identifying a set of data and a manner of processing the set of data, wherein the first query includes a set of commands;
process the first query to determine the set of data includes a dataset residing in a shared storage system and accessible via a second query system using a second query language that is different from the first query language;
translate the first query into a second query in the second query language by:
identifying a first query command in the first query language that has a corresponding query command in the second query language,
translating the first query command in the first query language to the second query language based on the first query command,
identifying a second query command in the first query language for which no corresponding query command exists in the second query language, and
generating a new query command in the second query language that performs functions similar to the second query command in the first query language; and
communicate the second query to a search coordinator of the second query system, wherein the search coordinator executes the second query on the dataset and generates search results.
11 . The non-transitory computer-readable media of claim 10 , wherein the second query command in the first query language comprises a time-based query command related to organizing, sorting, and/or selecting events based on an associated time or timestamp.
12 . A computing device of a first query system, the computing device comprising:
a data store; and
one or more processors, the one or more processors configured to:
receive, at the computing device of the first query system, a first query in a first query language, the first query identifying a set of data and a manner of processing the set of data, wherein the first query includes a set of commands;
process the first query to determine the set of data includes a dataset residing in a shared storage system and accessible via a second query system using a second query language that is different from the first query language;
translate the first query into a second query in the second query language by:
identifying a first query command in the first query language that has a corresponding query command in the second query language,
translating the first query command in the first query language to the second query language based on the first query command,
identifying a second query command in the first query for which no corresponding query command exists in the second query language, and
generating a new query command in the second query language that performs functions similar to the second query command in the first query language; and
communicate the second query to a search coordinator of the second query system, wherein the search coordinator executes the second query on the dataset and generates search results.
13 . The computing device of claim 12 , wherein the second query command in the first query language comprises a time-based query command related to organizing, sorting, and/or selecting events based on an associated time or timestamp.
14 . The computing device of claim 12 , wherein generating the new query command in the second query language comprises using one or more user query parameters to determine which and how many query commands or system query parameters to generate in the second query language.
15 . The computing device of claim 12 , wherein generating the new query command in the second query language comprises generating one or more query statements, query commands, system query parameters, lines of a query, or a combination thereof, in the second query language.
16 . The computing device of claim 12 , wherein the first query system and the search coordinator of the second query system are instantiated in a shared computing resource environment of a cloud service provider and are associated with a first cloud service account of the cloud service provider,
wherein the dataset stored in a first location comprises a first storage system of the cloud service provider and is associated with a second cloud service account of the cloud service provider.
17 . The computing device of claim 12 , wherein the one or more processors are further configured to:
determining a location of the search results;
determining a quantity of the search results;
generating a plurality of results readers based on the determined quantity of the search results;
concurrently retrieving the search results from the determined location using the generated plurality of results readers;
processing the search results to determine a set of search results; and
communicating the set of search results to a second computing device associated with a user.
18 . The computing device of claim 17 , wherein processing the search results comprises changing a format of the search results to a second format to generate the set of search results, wherein communicating the set of search results comprises communicating the set of search results in the second format.
19 . The computing device of claim 17 , wherein processing the search results comprises combining the search results with additional search results received from a third query system to determine the set of search results.
20 . The computing device of claim 12 , wherein the first query system and the search coordinator of the second query system are instantiated in a shared computing resource environment of a cloud service provider and are associated with a first cloud service account of the cloud service provider, and
wherein the dataset stored in a first location comprises a portion of the shared storage system of the cloud service provider and is associated with a second cloud service account of the cloud service provider.