Interleaved execution infrastructure in database engines
Interleaved infrastructure can interleave fragment execution during query compilation. For example, compilation can be paused, and a plan fragment is executed. The results of the plan fragment can be fed back into the compiler, which can then restart the compilation and modify the compilation based on the results of the plan fragment execution. The interleaved workflow provides a flexible, robust infrastructure, which can enable and improve a variety of database features and query optimizations.
1 . A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving, by a network-based data system, a query;
generating, by at least one server in the network-based data system, a query plan to execute the query;
detecting a plan fragment of the query plan for early execution, the plan fragment comprising a dynamic resolving object, the dynamic resolving object comprising a macro expression;
pausing compilation of the query plan based on detecting the plan fragment;
assigning, by the at least one server, the plan fragment to one or more execution platforms in the network-based data system for execution;
executing, by one or more execution platforms in the network-based data system, the plan fragment while the compilation of the query plan is paused, the executing comprising:
expanding a definition of a user defined function (UDF) in the plan fragment;
executing the macro expression based on the expanded definition; and
generating results based on executing the macro expression, the results comprising at least one generated definition of the UDF;
receiving, by the at least one server, results of executing the plan fragment from the one or more execution platforms;
modifying, by the at least one server, the compilation of the query plan based on the results; and
completing compilation of the query plan.
2 . The system of claim 1 , wherein the plan fragment further comprises a dynamic evaluation expression.
3 . The system of claim 2 , wherein the dynamic evaluation expression comprises a constant subquery.
4 . The system of claim 1 , the operations further comprising:
collecting query plan properties of the query plan, comprising a list of classes referenced in the query plan;
wherein detecting the plan fragment is based on the query plan properties.
5 . The system of claim 1 , wherein executing the plan fragment comprises executing a recursive query based on the plan fragment.
6 . A method comprising:
receiving, by a network-based data system, a query;
generating, by at least one server in the network-based data system, a query plan to execute the query;
detecting a plan fragment of the query plan for early execution, the plan fragment comprising a dynamic resolving object, the dynamic resolving object comprising a macro expression;
pausing compilation of the query plan based on detecting the plan fragment;
assigning, by the at least one server, the plan fragment to one or more execution platforms in the network-based data system for execution;
executing, by one or more execution platforms in the network-based data system, the plan fragment while the compilation of the query plan is paused, the executing comprising:
expanding a definition of a user defined function (UDF) in the plan fragment;
executing the macro expression based on the expanded definition; and
generating results based on executing the macro expression, the results comprising at least one generated definition of the UDF;
receiving, by the at least one server, results of executing the plan fragment from the one or more execution platforms;
modifying, by the at least one server, the compilation of the query plan based on the results; and
completing compilation of the query plan.
7 . The method of claim 6 , wherein the plan fragment further comprises a dynamic evaluation expression.
8 . The method of claim 7 , wherein the dynamic evaluation expression comprises a constant subquery.
9 . The method of claim 6 , further comprising:
collecting query plan properties of the query plan, comprising a list of classes referenced in the query plan;
wherein detecting the plan fragment is based on the query plan properties.
10 . The method of claim 6 , wherein executing the plan fragment comprises executing a recursive query based on the plan fragment.
11 . A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, by a network-based data system, a query;
generating, by at least one server in the network-based data system, a query plan to execute the query;
detecting a plan fragment of the query plan for early execution, the plan fragment comprising a dynamic resolving object, the dynamic resolving object comprising a macro expression;
pausing compilation of the query plan based on detecting the plan fragment;
assigning, by the at least one server, the plan fragment to one or more execution platforms in the network-based data system for execution;
executing, by one or more execution platforms in the network-based data system, the plan fragment while the compilation of the query plan is paused, the executing comprising:
expanding a definition of a user defined function (UDF) in the plan fragment;
executing the macro expression based on the expanded definition; and
generating results based on executing the macro expression, the results comprising at least one generated definition of the UDF;
receiving, by the at least one server, results of executing the plan fragment from the one or more execution platforms;
modifying, by the at least one server, the compilation of the query plan based on the results; and
completing compilation of the query plan.
12 . The machine-storage medium of claim 11 , wherein the plan fragment further comprises a dynamic evaluation expression.
13 . The machine-storage medium of claim 12 , wherein the dynamic evaluation expression comprises a constant subquery.
14 . The machine-storage medium of claim 11 , the operations further comprising:
collecting query plan properties of the query plan, comprising a list of classes referenced in the query plan;
wherein detecting the plan fragment is based on the query plan properties.