Interleaved execution infrastructure in database engines
Interleaved infrastructure can interleave fragment execution during query compilation. For example, compilation can be paused, and a plan fragment is executed. The results of the plan fragment can be fed back into the compiler, which can then restart the compilation and modify the compilation based on the results of the plan fragment execution. The interleaved workflow provides a flexible, robust infrastructure, which can enable and improve a variety of database features and query optimizations.
1. A method comprising:
receiving, by a network-based data system, a query;
generating, by at least one server in the network-based data system, a query plan to execute the query;
collecting query plan properties of the query plan, including a list of classes referenced in the query plan;
initiating, by the at least one server, compilation of the query plan;
identifying a plan fragment of the query plan for execution based on the query plan properties;
pausing compilation of the query plan;
assigning, by the at least one server, one or more jobs associated with the plan fragment to one or more execution platforms in the network-based data system;
executing the plan fragment while the compilation of the query plan is paused, including executing the one or more jobs by the one or more execution platforms;
generating a plurality of sampling query plans from the plan fragment on top of respective table scans for a plurality of tables stored in the network-based data system;
executing a plurality of sampling queries in parallel based on the plurality of sampling query plans;
retrieving statistics based on executing the plurality of sampling queries, the statistics including filter selectivity information of the plurality of tables;
receiving, by the at least one server, results of executing the plan fragment and the statistics; and
resuming the compilation of the query plan, the resuming of the compilation including modifying the compilation of the query plan based on the received results and the statistics including the filter selectivity information.
2. The method of claim 1 , wherein identifying the plan fragment includes detecting a dynamic resolving object in the query plan.
3. The method of claim 2 , wherein the dynamic resolving object includes a macro expression.
4. The method of claim 1 , wherein identifying the plan fragment includes detecting a dynamic evaluation expression.
5. The method of claim 4 , wherein the dynamic evaluation expression includes a constant subquery.
6. The method of claim 1 , wherein identifying the plan fragment includes detecting that the plan fragment is eligible for parallel execution.
7. The method of claim 6 , wherein the parallel execution includes parallel pruning.
8. The method of claim 1 , wherein identifying the plan fragment includes detecting use of information for optimizing the query plan.
9. The method of claim 1 , wherein executing the plan fragment includes executing a recursive query based on the plan fragment.
10. A non-transitory machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, by a network-based data system, a query;
generating, by at least one server in the network-based data system, a query plan to execute the query;
collecting query plan properties of the query plan, including a list of classes referenced in the query plan;
initiating, by the at least one server, compilation of the query plan;
identifying a plan fragment of the query plan for execution based on the query plan properties;
pausing compilation of the query plan;
assigning, by the at least one server, one or more jobs associated with the plan fragment to one or more execution platforms in the network-based data system;
executing the plan fragment while the compilation of the query plan is paused, including executing the one or more jobs by the one or more execution platforms;
generating a plurality of sampling query plans from the plan fragment on top of respective table scans for a plurality of tables stored in the network-based data system;
executing a plurality of sampling queries in parallel based on the plurality of sampling query plans;
retrieving statistics based on executing the plurality of sampling queries, the statistics including filter selectivity information of the plurality of tables;
receiving, by the at least one server, results of executing the plan fragment and the statistics; and
resuming the compilation of the query plan, the resuming of the compilation including modifying the compilation of the query plan based on the received results and the statistics including the filter selectivity information.
11. The non-transitory machine-storage medium of claim 10 , wherein identifying the plan fragment includes detecting a dynamic resolving object in the query plan.
12. The non-transitory machine-storage medium of claim 11 , wherein the dynamic resolving object includes a macro expression.
13. The non-transitory machine-storage medium of claim 10 , wherein identifying the plan fragment includes detecting a dynamic evaluation expression.
14. The non-transitory machine-storage medium of claim 13 , wherein the dynamic evaluation expression includes a constant subquery.
15. The non-transitory machine-storage medium of claim 10 , wherein identifying the plan fragment includes detecting that the plan fragment is eligible for parallel execution.
16. The non-transitory machine-storage medium of claim 15 , wherein the parallel execution includes parallel pruning.
17. The non-transitory machine-storage medium of claim 10 , wherein identifying the plan fragment includes detecting use of information for optimizing the query plan.
18. The non-transitory machine-storage medium of claim 10 , wherein executing the plan fragment includes executing a recursive query based on the plan fragment.
19. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving, by a network-based data system, a query;
generating, by at least one server in the network-based data system, a query plan to execute the query;
collecting query plan properties of the query plan, including a list of classes referenced in the query plan;
initiating, by the at least one server, compilation of the query plan;
identifying a plan fragment of the query plan for execution based on the query plan properties;
pausing compilation of the query plan;
assigning, by the at least one server, one or more jobs associated with the plan fragment to one or more execution platforms in the network-based data system;
executing the plan fragment while the compilation of the query plan is paused, including executing the one or more jobs by the one or more execution platforms;
generating a plurality of sampling query plans from the plan fragment on top of respective table scans for a plurality of tables stored in the network-based data system;
executing a plurality of sampling queries in parallel based on the plurality of sampling query plans;
retrieving statistics based on executing the plurality of sampling queries, the statistics including filter selectivity information of the plurality of tables;
receiving, by the at least one server, results of executing the plan fragment and the statistics; and
resuming the compilation of the query plan, the resuming of the compilation including modifying the compilation of the query plan based on the received results and the statistics including the filter selectivity information.
20. The system of claim 19 , wherein identifying the plan fragment includes detecting a dynamic resolving object in the query plan.
21. The system of claim 20 , wherein the dynamic resolving object includes a macro expression.
22. The system of claim 19 , wherein identifying the plan fragment includes detecting a dynamic evaluation expression.
23. The system of claim 22 , wherein the dynamic evaluation expression includes a constant subquery.
24. The system of claim 19 , wherein identifying the plan fragment includes detecting that the plan fragment is eligible for parallel execution.
25. The system of claim 24 , wherein the parallel execution includes parallel pruning.
26. The system of claim 19 , wherein identifying the plan fragment includes detecting use of information for optimizing the query plan.
27. The system of claim 19 , wherein executing the plan fragment includes executing a recursive query based on the plan fragment.