Implementing contract-based polymorphic and parallelizable SQL user-defined scalar and aggregate functions
Disclosed are systems and methods for implementing contract-based polymorphic and parallelizable user-defined scalar and aggregate functions. The systems and methods can include receiving a query including a plurality of user-defined functions, parsing the query into a plurality of nodes (e.g., basic operation unit or atomic operator), generating an execution plan that minimizes data transfer between the plurality of nodes, and executing the plan in a distributed environment. Each of the plurality of user-defined functions can correspond to one of a plurality of nodes.
1. A method comprising:
receiving, at a computing device including a processor, a query including a plurality of dynamically polymorphic user-defined functions, the plurality of dynamically polymorphic user defined functions including a user-defined scalar function and a user-defined aggregate function and each of the plurality of dynamically polymorphic user-defined functions includes a runtime contract that specifies acceptable input types and corresponding output types for each of the user-defined functions;
parsing, by the computing device, the query into a plurality of nodes, each of the plurality of user-defined functions corresponding to one of a plurality of nodes;
determining, by the computing device, input and output schemas for said plurality of dynamically polymorphic user-defined functions in accordance with said runtime contracts;
generating, by the computing device, an execution plan that minimizes data transfer between the plurality of nodes; and
storing, by the computing device, the execution plan to a memory.
2. The method of claim 1 , wherein generating the execution plan includes applying heuristic rules to determine runtimes and the data transfer between a plurality of execution plan fragments.
3. The method of claim 1 , wherein generating the execution plan includes combining two of the dynamically polymorphic user-defined functions into one operator that can be executed in one local path of data in a worker node.
4. The method of claim 1 , further comprising executing a component of the execution plan in a corresponding node of the plurality of nodes.
5. A system comprising:
a processor; and
a memory that stores instructions that, when execute by the processor, cause the processor to perform operations comprising:
receiving a query including a plurality of dynamically polymorphic user-defined functions, the plurality of dynamically polymorphic user defined functions including a user-defined scalar function and a user-defined aggregate function and each of the plurality of dynamically polymorphic user-defined functions includes a runtime contract that specifies acceptable input types and corresponding output types for each of the user-defined functions,
parsing the query into a plurality of nodes, each of the plurality of user-defined functions corresponding to one of a plurality of nodes,
determining input and output schemas for said plurality of dynamically polymorphic user-defined functions in accordance with said runtime contracts;
generating an execution plan that minimizes data transfer between the plurality of nodes, and
storing the execution plan to the memory.
6. The system of claim 5 , wherein generating the execution plan includes applying heuristic rules to determine runtimes and the data transfer between a plurality of execution plan fragments.
7. The system of claim 5 , wherein generating the execution plan includes combining two of the dynamically polymorphic user-defined functions into one operator that can be executed in one local path of data in a worker node.
8. The system of claim 5 , wherein the operations further comprise executing a component of the execution plan in a corresponding node of the plurality of nodes.
9. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving a query including a plurality of dynamically polymorphic user-defined functions, the plurality of dynamically polymorphic user defined functions including a user-defined scalar function and a user-defined aggregate function and each of the plurality of dynamically polymorphic user-defined functions includes a runtime contract that specifies acceptable input types and corresponding output types for each of the user-defined functions,
parsing the query into a plurality of nodes, each of the plurality of user-defined functions corresponding to one of a plurality of nodes,
determining input and output schemas for said plurality of dynamically polymorphic user-defined functions in accordance with said runtime contracts;
generating an execution plan that minimizes data transfer between the plurality of nodes, and
storing the execution plan to the memory.
10. The non-transitory computer-readable medium of claim 9 , wherein generating the execution plan includes applying heuristic rules to determine runtimes and the data transfer between a plurality of execution plan fragments.
11. The non-transitory computer-readable medium of claim 9 , wherein generating the execution plan includes combining two of the dynamically polymorphic user-defined functions into one operator that can be executed in one local path in a worker node.
12. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise executing a component of the execution plan in a corresponding node of the plurality of nodes.