Accessing files in a database stage using a user defined function
A file access system for user defined functions (UDFs) can be implemented on a distributed database system. The system can store UDF interfaces and file reference objects that can be called by other users. Upon a UDF being called, files on a stage, one or more interface objects (e.g., InputStream), and file reference objects can be implemented by execution nodes of the distributed database system. The execution nodes can implement multiple threads that are authenticated and can download file data from a staging location concurrently.
1. A method comprising:
storing, on a database, a user defined function file comprising a function and a file reference object that passes a file path to a class of the function such that the only path for files used by the function is a path as specified in a database call from a database user;
receiving the database call for the function, the database call comprising the path to one or more files;
activating the function using the path passed into the class from the received database call;
loading the one or more files on a node of the database using the file path as specified in the database call;
generating, by one or more hardware processors, result data by processing the one or more files using the function; and
storing the result data.
2. The method of claim 1 , wherein the database user defined function file is called by specifying a function name in the database call of the database user, where the function name is associated by the database with the class that comprises the function.
3. The method of claim 1 , further comprising:
creating a storage location reference to a storage location on the database, wherein the storage location reference is created according to a create storage location request generated on a database user account of the database.
4. The method of claim 1 , wherein the path to the one or more files of the received database call is a storage path to a data location on an external storage device that is external to the database.
5. The method of claim 1 , wherein the database call is received by a function server of the database and transmitted to the node using a network channel.
6. The method of claim 5 , wherein the network channel comprises a remote procedure call (RPC) channel.
7. The method of claim 6 , wherein the user defined function file and the function are activated in a runtime environment on the node.
8. The method of claim 1 , wherein the user defined function file is generated by a provider database of the database.
9. The method of claim 8 , wherein the node is managed by a consumer database of the database that is distinct from the provider database of the database.
10. The method of claim 9 , wherein the function is activated in a runtime environment in the node of the consumer database according to the user defined function file that is generated by the provider database.
11. The method of claim 10 , wherein the user defined function file comprises a Java Application Programming (JAR) file.
12. The method of claim 10 , wherein the runtime environment comprises a Java Virtual Machines (JVM).
13. A system comprising:
one or more processors of a machine; and
at least one memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
storing, on a database, a user defined function file comprising a function and a file reference object that passes a file path to a class of the function such that the only path for files used by the function is a path as specified in a database call from a database user;
receiving the database call for the function, the database call comprising the path to one or more files;
activating the function using the path passed into the class from the received database call;
loading the one or more files on a node of the database using the file path as specified in the database call;
generating, by one or more hardware processors, result data by processing the one or more files using the function; and
storing the result data.
14. The system of claim 13 , wherein the user defined function file is called by specifying a function name in the database call of the user, where the function name is associated by the database with the class that comprises the function.
15. The system of claim 13 , the operations further comprising:
creating a storage location reference to a storage location on the database, wherein the storage location reference is created according to a create storage location request generated on a database user account of the database.
16. The system of claim 13 , wherein the path to the one or more files of the received database call is a storage path to a data location on an external storage device that is external to the database.
17. The system of claim 13 , wherein the database call is received by a function server of the database and transmitted to the node using a network channel.
18. The system of claim 17 , wherein the network channel comprises a remote procedure call (RPC) channel.
19. The system of claim 18 , wherein the database user defined function file and the function are activated in a runtime environment on the node.
20. The system of claim 13 , wherein the user defined function file is generated by a provider database of the database.
21. The system of claim 20 , wherein the node is managed by a consumer database of the database that is distinct from the provider database of the database.
22. The system of claim 21 , wherein the function is activated in a runtime environment in the node of the consumer database according to the user defined function file that is generated by the provider database.
23. The system of claim 22 , wherein the user defined function file comprises a Java Application Programming (JAR) file.
24. The system of claim 22 , wherein the runtime environment comprises a Java Virtual Machines (JVM).
25. A machine storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
storing, on a database, a user defined function file comprising a function and a file reference object that passes a file path to a class of the function such that the only path for files used by the function is a path as specified in a database call from a database user;
receiving the database call for the function, the database call comprising the path to one or more files;
activating the function using the path passed into the class from the received database call;
loading the one or more files on a node of the database using the file path as specified in the database call;
generating, by one or more hardware processors, result data by processing the one or more files using the function; and
storing the result data.
26. The machine storage medium of claim 25 , wherein the user defined function file is called by specifying a function name in the database call of the database user, where the function name is associated by the database with the class that comprises the function.
27. The machine storage medium of claim 25 , the operations further comprising:
creating a storage location reference to a storage location on the database, wherein the storage location reference is created according to a create storage location request generated on a database user account of the database.
28. The machine storage medium of claim 25 , wherein the path to the one or more files of the received database call is a storage path to a data location on an external storage device that is external to the database.
29. The machine storage medium of claim 25 , wherein the database call is received by a function server of the database and transmitted to the node using a network channel.
30. The machine storage medium of claim 29 , wherein the network channel comprises a remote procedure call (RPC) channel.