Inline compilation of user defined functions
Embodiments described herein provide techniques for in-line compiling of UDFs in other programming languages. These techniques enable faster programming iterations because it allows users to compile directly in the cloud processing system. Moreover, it allows the UDFs to tie into existing libraries. The compiled results are treated as sensitive and handled with appropriate security policies, as with any other user data in the system.
1. A method comprising:
receiving, by a cloud processing system operating using a first programming language, a query referencing a user defined function (UDF) including a body with a code portion in a second programming language;
generating a query plan to execute the query, the query plan including at least one function associated with the UDF;
transmitting the query plan to a processing node;
providing a worker process of the processing node and providing a separate UDF server in the processing node, the UDF server and the worker process being associated with separate processors of the processing node;
transmitting the code portion of the UDF from the worker process to the UDF server via a remote call protocol, the UDF server being restricted from accessing select data accessible by the worker process;
compiling the code portion of the UDF to generate compiled results in a language-specific integration operating on the UDF server, the UDF server operating using the first programming language and the language-specific integration operating in the second programming language; and
transmitting the compiled results from the UDF server to the worker process via the remote call protocol,
generating a response to the query based on the compiled results.
2. The method of claim 1 , wherein the query plan includes a plan tree with a plurality of functions and the at least one function associated with the UDF includes a select function.
3. The method of claim 1 , wherein a compute service manager of the cloud processing system receives the query.
4. The method of claim 1 , wherein the second programming language is JAVA.
5. The method of claim 1 , wherein the second programming language is C Sharp.
6. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, by a cloud processing system operating using a first programming language, a query referencing a user defined function (UDF) including a body with a code portion in a second programming language;
generating a query plan to execute the query, the query plan including at least one function associated with the UDF;
transmitting the query plan to a processing node;
providing a worker process of the processing node and providing a separate UDF server in the processing node, the UDF server and the worker process being associated with separate processors of the processing node;
transmitting the code portion of the UDF from the worker process to the UDF server via a remote call protocol, the UDF server being restricted from accessing select data accessible by the worker process;
compiling the code portion of the UDF to generate compiled results in a language-specific integration operating on the UDF server, the UDF server operating using the first programming language and the language-specific integration operating in the second programming language; and
transmitting the compiled results from the UDF server to the worker process via the remote call protocol,
generating a response to the query based on the compiled results.
7. The machine-storage medium of claim 6 , wherein the query plan includes a plan tree with a plurality of functions and the at least one function associated with the UDF includes a select function.
8. The machine-storage medium of claim 6 , wherein a compute service manager of the cloud processing system receives the query.
9. The machine-storage medium of claim 6 , wherein the second programming language is JAVA.
10. The machine-storage medium of claim 6 , wherein the second programming language is C Sharp.
11. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving, by a cloud processing system operating using a first programming language, a query referencing a user defined function (UDF) including a body with a code portion in a second programming language;
generating a query plan to execute the query, the query plan including at least one function associated with the UDF;
transmitting the query plan to a processing node;
providing a worker process of the processing node and providing a separate UDF server in the processing node, the UDF server and the worker process being associated with separate processors of the processing node;
transmitting the code portion of the UDF from the worker process to the UDF server via a remote call protocol, the UDF server being restricted from accessing select data accessible by the worker process;
compiling the code portion of the UDF to generate compiled results in a language-specific integration operating on the UDF server, the UDF server operating using the first programming language and the language-specific integration operating in the second programming language; and
transmitting the compiled results from the UDF server to the worker process via the remote call protocol,
generating a response to the query based on the compiled results.
12. The system of claim 11 , wherein the query plan includes a plan tree with a plurality of functions and the at least one function associated with the UDF includes a select function.
13. The system of claim 11 , wherein a compute service manager of the cloud processing system receives the query.
14. The system of claim 11 , wherein the second programming language is JAVA.
15. The system of claim 11 , wherein the second programming language is C Sharp.