Scalable query processing
Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.
1. A method comprising:
generating a query plan to execute a query directed to a data set;
determining that a portion of the query plan is eligible for fragment processing based on a set of criteria;
executing, by a parent query coordinator using one or more computing resources of a first set of computing resources assigned to the parent query coordinator, the identified portion of the query plan on a first batch of files of the data set from a continuous scanset to generate a first batch result;
transmitting instructions to a fragment query coordinator for the fragment query coordinator to execute the identified portion of the query on a second batch of files of the data set from the continuous scanset to generate a materialized result file using one or more fragment computing resources of a second set of computing resources;
after all files in the continuous scanset are processed by the parent query coordinator and fragment query coordinator, scanning, by the parent query coordinator, the materialized result file to generate scanned results;
combining the first batch results and the scanned results to generate combined results; and
executing remaining portion of the query plan on the combined results to generate a response to the query.
2. The method of claim 1 , further comprising:
loading files of the data set into a first shared file queue as the continuous scanset;
grouping a first set of files as the first batch and providing the first batch to the parent query coordinator; and
grouping a second set of files as the second batch and providing the second batch to the fragment query coordinator.
3. The method of claim 2 , further comprising:
providing additional batches serially until all files in the continuous scanset have been provided.
4. The method of claim 2 , wherein the fragment query coordinator loads the materialized result file in an output shared file queue.
5. The method of claim 1 , wherein the set of criteria includes whether an output of execution of the identified portion is less than an input of the identified portion.
6. The method of claim 1 , wherein the set of criteria includes whether the identified portion is executable by a computing resource without communicating with another computing resource.
7. The method of claim 6 , wherein the query plan includes a plurality of operators and links connecting the operators, each link connecting a first and second operator of the plurality of operators and indicating whether the first operator is executable by a computing resource without communicating with another computing resource.
8. The method of claim 1 , further comprising:
aggregating the materialized result file with at least another materialized result file; and
wherein scanning the materialized result file to generate scanned results includes scanning the aggregated materialized result files to generate the scanned results.
9. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
generating a query plan to execute a query directed to a data set;
determining that a portion of the query plan is eligible for fragment processing based on a set of criteria;
executing, by a parent query coordinator using one or more computing resources of a first set of computing resources assigned to the parent query coordinator, the identified portion of the query plan on a first batch of files of the data set from a continuous scanset to generate a first batch result;
transmitting instructions to a fragment query coordinator for the fragment query coordinator to execute the identified portion of the query on a second batch of files of the data set from the continuous scanset to generate a materialized result file using one or more fragment computing resources of a second set of computing resources;
after all files in the continuous scanset are processed by the parent query coordinator and fragment query coordinator, scanning, by the parent query coordinator, the materialized result file to generate scanned results;
combining the first batch results and the scanned results to generate combined results; and
executing remaining portion of the query plan on the combined results to generate a response to the query.
10. The system of claim 9 , the operations further comprising:
loading files of the data set into a first shared file queue as the continuous scanset;
grouping a first set of files as the first batch and providing the first batch to the parent query coordinator; and
grouping a second set of files as the second batch and providing the second batch to the fragment query coordinator.
11. The system of claim 10 , the operations further comprising:
providing additional batches serially until all files in the continuous scanset have been provided.
12. The system of claim 10 , wherein the fragment query coordinator loads the materialized result file in an output shared file queue.
13. The system of claim 9 , wherein the set of criteria includes whether an output of execution of the identified portion is less than an input of the identified portion.
14. The system of claim 9 , wherein the set of criteria includes whether the identified portion is executable by a computing resource without communicating with another computing resource.
15. The system of claim 14 , wherein the query plan includes a plurality of operators and links connecting the operators, each link connecting a first and second operator of the plurality of operators and indicating whether the first operator is executable by a computing resource without communicating with another computing resource.
16. The system of claim 1 , the operations further comprising:
aggregating the materialized result file with at least another materialized result file; and
wherein scanning the materialized result file to generate scanned results includes scanning the aggregated materialized result files to generate the scanned results.
17. A non-transitory machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
generating a query plan to execute a query directed to a data set;
determining that a portion of the query plan is eligible for fragment processing based on a set of criteria;
executing, by a parent query coordinator using one or more computing resources of a first set of computing resources assigned to the parent query coordinator, the identified portion of the query plan on a first batch of files of the data set from a continuous scanset to generate a first batch result;
transmitting instructions to a fragment query coordinator for the fragment query coordinator to execute the identified portion of the query on a second batch of files of the data set from the continuous scanset to generate a materialized result file using one or more fragment computing resources of a second set of computing resources;
after all files in the continuous scanset are processed by the parent query coordinator and fragment query coordinator, scanning, by the parent query coordinator, the materialized result file to generate scanned results;
combining the first batch results and the scanned results to generate combined results; and
executing remaining portion of the query plan on the combined results to generate a response to the query.
18. The non-transitory machine-storage medium of claim 17 , further comprising:
loading files of the data set into a first shared file queue as the continuous scanset;
grouping a first set of files as the first batch and providing the first batch to the parent query coordinator; and
grouping a second set of files as the second batch and providing the second batch to the fragment query coordinator.
19. The non-transitory machine-storage medium of claim 18 , further comprising:
providing additional batches serially until all files in the continuous scanset have been provided.
20. The non-transitory machine-storage medium of claim 18 , wherein the fragment query coordinator loads the materialized result file in an output shared file queue.
21. The non-transitory machine-storage medium of claim 17 , wherein the set of criteria includes whether an output of execution of the identified portion is less than an input of the identified portion.
22. The non-transitory machine-storage medium of claim 17 , wherein the set of criteria includes whether the identified portion is executable by a computing resource without communicating with another computing resource.
23. The non-transitory machine-storage medium of claim 22 , wherein the query plan includes a plurality of operators and links connecting the operators, each link connecting a first and second operator of the plurality of operators and indicating whether the first operator is executable by a computing resource without communicating with another computing resource.
24. The non-transitory machine-storage medium of claim 17 , further comprising:
aggregating the materialized result file with at least another materialized result file; and
wherein scanning the materialized result file to generate scanned results includes scanning the aggregated materialized result files to generate the scanned results.