Inline compilation of user defined functions
Embodiments described herein provide techniques for in-line compiling of UDFs in other programming languages. These techniques enable faster programming iterations because it allows users to compile directly in the cloud processing system. Moreover, it allows the UDFs to tie into existing libraries. The compiled results are treated as sensitive and handled with appropriate security policies, as with any other user data in the system.
1. A method comprising:
receiving, by a compute service manager of a cloud processing system operating using a first programming language, a query referencing a user defined function (UDF) including a body with a code portion in a second programming language;
generating a query plan to execute the query, the query plan including at least one function associated with the UDF;
transmitting the query plan to a processing node;
retrieving, by the processing node, stored compiled results associated with the UDF, wherein the processing node references a target storage defined in the UDF to retrieve the stored compiled results;
executing the UDF with the compiled results using data in a database referenced in the query; and
generating a response to the query based on executing the UDF.
2. The method of claim 1 , wherein the query plan includes a plan tree with a plurality of functions and the at least one function associated with the UDF includes a select function.
3. The method of claim 1 , further comprising:
decrypting the compiled results based on encryption information defined in the UDF.
4. The method of claim 1 , wherein the second programming language is JAVA.
5. The method of claim 1 , wherein the second programming language is C Sharp.
6. The method of claim 1 , wherein the compiled results are generated using a UDF server in the processing node separate from a worker process of the processing node,
wherein the UDF server and the worker process are associated with separate processors of the processing node and wherein the UDF server is restricted from accessing select data accessible by the worker process.
7. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, by a compute service manager of a cloud processing system operating using a first programming language, a query referencing a user defined function (UDF) including a body with a code portion in a second programming language;
generating a query plan to execute the query, the query plan including at least one function associated with the UDF;
transmitting the query plan to a processing node;
retrieving, by the processing node, stored compiled results associated with the UDF, wherein the processing node references a target storage defined in the UDF to retrieve the stored compiled results;
executing the UDF with the compiled results using data in a database referenced in the query; and
generating a response to the query based on executing the UDF.
8. The machine-storage medium of claim 7 , wherein the query plan includes a plan tree with a plurality of functions and the at least one function associated with the UDF includes a select function.
9. The machine-storage medium of claim 7 , further comprising:
decrypting the compiled results based on encryption information defined in the UDF.
10. The machine-storage medium of claim 7 , wherein the second programming language is JAVA.
11. The machine-storage medium of claim 7 , wherein the second programming language is C Sharp.
12. The machine-storage medium of claim 7 , wherein the compiled results are generated using a UDF server in the processing node separate from a worker process of the processing node,
wherein the UDF server and the worker process are associated with separate processors of the processing node and wherein the UDF server is restricted from accessing select data accessible by the worker process.
13. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving, by a compute service manager of a cloud processing system operating using a first programming language, a query referencing a user defined function (UDF) including a body with a code portion in a second programming language;
generating a query plan to execute the query, the query plan including at least one function associated with the UDF;
transmitting the query plan to a processing node;
retrieving, by the processing node, stored compiled results associated with the UDF, wherein the processing node references a target storage defined in the UDF to retrieve the stored compiled results;
executing the UDF with the compiled results using data in a database referenced in the query; and
generating a response to the query based on executing the UDF.
14. The system of claim 13 , wherein the query plan includes a plan tree with a plurality of functions and the at least one function associated with the UDF includes a select function.
15. The system of claim 13 , the operations further comprising:
decrypting the compiled results based on encryption information defined in the UDF.
16. The system of claim 13 , wherein the second programming language is JAVA.
17. The system of claim 13 , wherein the second programming language is C Sharp.
18. The system of claim 13 , wherein the compiled results are generated using a UDF server in the processing node separate from a worker process of the processing node,
wherein the UDF server and the worker process are associated with separate processors of the processing node and wherein the UDF server is restricted from accessing select data accessible by the worker process.