Accessing files in a database stage using a user defined function
A file access system for user defined functions (UDFs) can be implemented on a distributed database system. The system can store UDF signatures and interfaces (e.g., classes, sub-classes) that can be called by other users. Upon a UDF being called, one or more interface objects (e.g., InputStream) can be created and requests transferred to a execution node via a network channel. The execution node can implement multiple threads that are authorized and download file data from a staging location (e.g., internal stage, external stage) concurrently.
1. A method comprising:
receiving, in a first database of a network database system, a plurality of database function definitions, a database function definition of the plurality of database function definitions, the database function definition specifying one or more input data streams and including a database function class type and a function name for a function to be executed in a second database of the network database system;
receiving, by the network database system, a database function including code related to at least one operation to be performed using the database function class type and the one or more input data streams specified by the database function definition, the database function having a function name that matches one of the plurality of database function definitions;
receiving a request to perform the database function, the request generated by the second database of the network database system;
activating, on a user defined function server, one or more objects of the database function class type that correspond to the requested database function, the one or more objects comprising one or more user defined function database objects;
retrieving, by the network database system, files specified in the request using the one or more input data streams and a path to a file storage stage that corresponds to the one or more activated objects of the database function class type; and
storing the files on the network database system.
2. The method of claim 1 , further comprising:
generating, by a function server of the second database, results data by applying a query on the files stored on a node of the network database system, wherein the files are provided to the function server over a network channel in the node.
3. The method of claim 2 , wherein the network channel is a remote procedure call (RPC) channel.
4. The method of claim 1 , wherein the file storage stage corresponds to a external storage device that is external to the network database system.
5. The method of claim 1 , further comprising:
determining, by the network database system, each thread is authorized to access one of the files based on a user of the second database being previously granted shared access to the each of the one of the files by the first database.
6. The method of claim 1 , wherein the database function is a database code file, and the method further comprises:
processing, by the second database, the files according to one or more instructions in the code of the database code file.
7. The method of claim 6 , wherein the database code file is a Java Application Programming (JAR) file.
8. The method of claim 1 , wherein the one or more objects are Java Virtual Machines (JVMs).
9. The method of claim 1 , wherein the one or more objects are activated using a database query generated by the second database.
10. The method of claim 9 , wherein the database query comprises a select statement that includes the function name of the database function.
11. The method of claim 10 , wherein the each of the plurality of database function definitions is a shareable function that is generated on the network database system by the first database, wherein the first database shares access to the shareable function for execution on the second database.
12. The method of claim 1 , wherein the database function class type is an input stream class.
13. A system comprising:
one or more processors of a machine; and
at least one memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving, in a first database of a network database system, a plurality of database function definitions, a database function definition of the plurality of database function definitions, the database function definition specifying one or more input data streams and including a database function class type and a function name for a function to be executed in a second database of the network database system;
receiving, by the network database system, a database function including code related to at least one operation to be performed using the database function class type and the one or more input data streams specified by the database function definition, the database function having a function name that matches one of the plurality of database function definitions;
receiving a request to perform the database function, the request generated by the second database of the network database system;
activating, on a user defined function server, one or more objects of the database function class type that correspond to the requested database function, the one or more objects comprising one or more user defined function database objects;
retrieving, by the network database system, files specified in the request using the one or more input data streams and a path to a file storage stage that corresponds to the one or more activated objects of the database function class type; and
storing the files on the network database system.
14. The system of claim 13 , further comprising:
generating, by a function server of the second database, results data by applying a query on the files stored on a node of the network database system, wherein the files are provided to the function server over a network channel in the node.
15. The system of claim 14 , wherein the network channel is a remote procedure call (RPC) channel.
16. The system of claim 13 , wherein the file storage stage corresponds to a external storage device that is external to the network database system.
17. The system of claim 13 , further comprising:
determining, by the network database system, that each thread is authorized to access one of the files based on a user of the second database being previously granted shared access to the each of the one of the files by the first database.
18. The system of claim 13 , wherein the database function is a database code file, and the system further comprises:
processing, by the second database, the files according to one or more instructions in the code of the database code file.
19. The system of claim 18 , wherein the database code file is a Java Application Programming (JAR) file.
20. The system of claim 13 , wherein the one or more objects are Java Virtual Machines (JVMs).
21. The system of claim 13 , wherein the one or more objects are activated using a database query generated by the second database.
22. The system of claim 21 , wherein the database query comprises a select statement that includes the function name of the database function.
23. The system of claim 22 , wherein the each of the plurality of database function definitions is a shareable function that is generated on the network database system by the first database, wherein the first database shares access to the shareable function for execution on the second database.
24. The system of claim 13 , wherein the database function class type is an input stream class.
25. A machine storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving, in a first database of a network database system, a plurality of database function definitions, a database function definition of the plurality of database function definitions, the database function definition specifying one or more input data streams and including a database function class type and a function name for a function to be executed in a second database of the network database system;
receiving, by the network database system, a database function including code related to at least one operation to be performed using the database function class type and the one or more input data streams specified by the database function definition, the database function having a function name that matches one of the plurality of database function definitions;
receiving a request to perform the database function, the request generated by the second database of the network database system;
activating, on a user defined function server, one or more objects of the database function class type that correspond to the requested database function, the one or more objects comprising one or more user defined function database objects;
retrieving, by the network database system, files specified in the request using the one or more input data streams and a path to a file storage stage that corresponds to the one or more activated objects of the database function class type; and
storing the files on the network database system.
26. The machine storage medium of claim 25 , further comprising:
generating, by a function server of the second database, results data by applying a query on the files stored on a node of the network database system, wherein the files are provided to the function server over a network channel in the node.
27. The machine storage medium of claim 26 , wherein the network channel is a remote procedure call (RPC) channel.
28. The machine storage medium of claim 25 , wherein the file storage stage corresponds to an external storage device that is external to the network database system.
29. The machine storage medium of claim 25 , the operations further comprising: determining that each thread is authorized to access one of the files based on a user of the second database being previously granted shared access to the each of the one of the files by the first database.
30. The machine storage medium of claim 25 , wherein the database function is in a database code file, and wherein the second database processes the files according to one or more instructions in the code of the database code file.