Distribute execution of user-defined function
A user-defined function (UDF) is received. Then, execution of the UDF is distributed into a plurality of phases. Lastly, each of the phases is executed separately on a relational database including an aggregate projection that stores an aggregate value calculated from a table column of a table using an aggregate function.
1. A method, comprising:
receiving a query request including a user-defined function (UDF);
distributing, by a processor, execution of the UDF in the query request into a plurality of phases including a data load phase, a query phase, and a maintenance phase; and
executing, by the processor, each of the plurality of phases separately on a relational database including an aggregate projection that stores an aggregate value calculated from a table column of a table using an aggregate function, wherein each of the plurality of phases applies a different transformation to data in the relational database.
2. The method of claim 1 , wherein the executing includes:
extracting data at a local node during the data load phase, and
analyzing the extracted data at a remote node that includes the relational database during the query phase.
3. The method of claim 1 , wherein the data load phase relates to loading data into the aggregate projection corresponding to the table.
4. The method of claim 3 , wherein the executing includes only processing the data stored on a local node of a plurality of nodes of a system during the data load phase, and not segmenting the data during the data load phase.
5. The method of claim 1 , wherein
the query phase relates to accessing the aggregate projection to retrieve the aggregate value stored in the aggregate projection, and
the executing includes aggregating the data within a partition of the aggregate projection during the query phase.
6. The method of claim 5 , further comprising:
parsing the query request to identify an operator, an operation and the UDF.
7. The method of claim 1 , wherein
the maintenance phase relates to at least one of consolidating a read optimized store (ROS) container and purging a deleted record, and
the executing includes aggregating the data from any partitions of the aggregate projection during the maintenance phase to determine an updated merged result with fully aggregated data.
8. The method of claim 1 , further comprising:
returning a result after the execution in response to the query request,
wherein the aggregate projection corresponding to the table is updated each time data is loaded to the table.
9. The method of claim 1 , wherein
the UDF includes at least one of a transform function and a scalar function,
the scalar function returns a single value as an output based on a set of input values, and
the transform function receives an input set of values and maps the input set of values on to an output set of values.
10. The method of claim 1 , wherein
the aggregate projection includes another UDF that is tracked for a dependency, and
the dependency relates to dropping the table column of the table.
11. The method of claim 1 , wherein
each of the phases is to receive an input table and to generate an output table, and
a schema for the output table is independent of a schema for the input table for at least one of the phases.
12. A system, comprising:
an interface unit to parse a query to identify a user-defined function (UDF) in the query;
a distribution unit to distribute execution of the UDF in the query into a plurality of phases including a data load phase, a query phase, and a maintenance phase; and
an execution unit to execute each of the plurality of phases separately on a relational database that includes an aggregate projection that stores an aggregate value calculated from a table column of a table, wherein each of the plurality of phases applies a different transformation to data in the relational database.
13. The system of claim 12 , wherein the execution unit is to at least one of break down and gather data during the data load phase, and process the data during at least one of the query and maintenance phases.
14. The system of claim 12 , wherein the execution unit is to:
extract data at a local node of a plurality of nodes during the data load phase; and
analyze the extracted data at a remote node of the plurality of nodes that includes the relational database during the query phase.
15. The system of claim 12 , wherein, during the maintenance phase, the execution unit is to:
consolidate a read optimized store (ROS) container or purge a deleted record; and
aggregate data from any partitions of the aggregate projection to determine an updated merged result with fully aggregated data.
16. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor of a device, cause the processor to:
parse a query to identify a user-defined function in the query;
distribute execution of the UDF in the query into a plurality of phases including a data load phase, a query phase, and a maintenance phase; and
execute the query including the UDF in each of the plurality of phases separately on a relational database, wherein each of the plurality of phases applies a different transformation to data in the relational database.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions are executable to cause the processor to:
load the data into a least one of the table and the aggregate projection corresponding to the table during the data load phase, and
access the aggregate projection to retrieve the aggregate value stored in the aggregate projection during the query phase.
18. The non-transitory computer-readable storage medium of claim 16 , wherein, to execute the query, the instructions are executable to cause the processor to:
extract data at a local node of a plurality of nodes during the data load phase; and
analyze the data at a remote node of the plurality of nodes that includes the relational database during the query phase.
19. The non-transitory computer-readable storage medium of claim 18 , wherein, during the data load phase, the instructions are executable to cause the processor to: only process the data stored in the local node without segmenting the data.
20. The non-transitory computer-readable storage medium of claim 16 , wherein, during the maintenance phase, the instructions are executable to cause the processor to:
consolidate a read optimized store (ROS) container or purge a deleted record; and
aggregate data from any partitions of the aggregate projection to determine an updated merged result with fully aggregated data.