Processing a database query using a shared metadata store
A method and system for executing a query in parallel is disclosed. A master node may receive a query from a client and develop query plans from that query. The query plans may be forwarded to worker nodes for execution, and each query plan may be accompanied by query metadata. The metadata may be stored in a catalog on the master node.
1. A method for executing queries in a parallel processing database system, comprising:
receiving a query at a master node, the master node comprising a database catalog including metadata defining database objects;
in response to receiving the query at the master node, initiating a catalog server session, taking a snapshot of the metadata, and associating the snapshot of the metadata with the catalog server session, wherein a separate catalog server session is initiated and a separate snapshot of the metadata is taken for each query;
transmitting a query plan and query metadata to a worker node, wherein the query plan is based on the query, wherein the query metadata includes metadata for executing the query plan, wherein the query metadata includes database table definitions that define database objects, and wherein the query metadata is retrieved from the snapshot of the metadata associated with the catalog server session;
receiving, by the master node, a request for additional metadata that is required for the worker node to execute the query plan, wherein in response to a determination that the worker node requires the additional metadata for the worker node to execute the query plan, the worker node queries a parent in a tree structure of a plurality of worker nodes in the parallel processing database system for the additional metadata, wherein the parent node is a node between the master node and the worker node in relation to the tree structure;
communicating, to the worker node, the additional metadata that is required for the worker node to execute the query plan, wherein the additional metadata is retrieved from a same session as the catalog server session corresponding to the query;
executing the query plan on the worker node; and
returning, to the master node, a result associated with the execution of the query plan on the worker node.
2. The method of claim 1 , further comprising generating a plurality of query plans based on the query.
3. The method of claim 2 , further comprising transmitting the plurality query plans to a plurality of worker nodes.
4. The method of claim 3 , further comprising executing the plurality of query plans in parallel.
5. The method of claim 1 , further comprising storing the query metadata in a cache on the worker node.
6. The method of claim 5 , further comprising clearing the cache after executing the query plan.
7. The method of claim 5 , further comprising retrieving the query metadata from the cache while executing the query plan.
8. The method of claim 1 , further comprising transmitting a request for the additional metadata from the worker node to the master node.
9. The method of claim 8 , further comprising transmitting the additional metadata to the worker node.
10. The method of claim 1 , further comprising:
compiling the result associated with the execution of the query plan on the worker node with another result associated with the query; and
returning the compiled result and other result as a final query result to a client.
11. The method of claim 1 , wherein the query metadata includes one or more of a user defined database function, a system defined database function, a database view, and a database index.
12. The method of claim 1 , further comprising:
receiving another query at the master node;
in response to receiving the other query, initiating another catalog server session, taking, another snapshot of the metadata as the metadata existed when the other catalog server session is initiated, and associating the other snapshot of the metadata with the other catalog server session; and
transmitting another query plan and other query metadata to the worker node, wherein the other query metadata is retrieved from the snapshot of the metadata associated with the other catalog server session.
13. The method of claim 1 , further comprising:
in response to receiving the query plan and the query metadata, determining, by the worker node, that additional metadata is required for the worker node to execute the query plan;
in response to determining that additional metadata is required, requesting, by the worker node, the additional metadata; and
receiving the additional metadata, wherein the additional metadata is retrieved from the snapshot of the metadata associated with the catalog server session.
14. The method of claim 1 , wherein in the event that additional metadata is required in connection with the query, the additional metadata is obtained from the snapshot of the metadata corresponding to the query.
15. The method of claim 1 , wherein the querying a parent in the tree structure of a plurality of worker nodes comprises successively querying one or more parent nodes for the additional metadata before querying the master node for the additional metadata.
16. The method of claim 15 , wherein the successively querying one or more parent nodes for the additional metadata before querying the master node for the additional metadata comprises at least one of the one or more parent nodes forwarding a request for the additional metadata to another of the one or more parent nodes.
17. The method of claim 1 , wherein the request for additional metadata is communicated to the master node in response to the worker node determining that the worker node requires the additional metadata to execute the query plan.
18. A computer program product for executing queries in a parallel processing database system, comprising a non-transitory computer readable medium having program instructions embodied therein for:
receiving a query at a master node, the master node comprising a database catalog including metadata defining database objects;
in response to receiving the query at the master node, initiating a catalog server session, taking a snapshot of the metadata, and associating the snapshot of the metadata with the catalog server session, wherein a separate catalog server session is initiated and a separate snapshot of the metadata is taken for each query;
transmitting a query plan and query metadata to a worker node, wherein the query plan is based on the query, wherein the query metadata includes metadata for executing the query plan, wherein the query metadata includes database table definitions that define database objects, and wherein the query metadata is retrieved from the snapshot of the metadata associated with the catalog server session;
receiving, by the master node, a request for additional metadata that is required for the worker node to execute the query plan, wherein in response to a determination that the worker node requires the additional metadata for the worker node to execute the query plan, the worker node queries a parent in a tree structure of a plurality of worker nodes in the parallel processing database system for the additional metadata, wherein the parent node is a node between the master node and the worker node in relation to the tree structure;
communicating, to the worker node, the additional metadata that is required for the worker node to execute the query plan, wherein the additional metadata is retrieved from a same session as the catalog server session corresponding to the query;
executing the query plan on the worker node; and
returning, to the master node, a result associated with the execution of the query plan on the worker node.
19. The computer program of claim 18 , further comprising instructions for storing the query metadata in a cache on the worker node.
20. The computer program of claim 19 , further comprising instructions for clearing the cache after executing the query plan.
21. The computer program of claim 19 , further comprising instructions for retrieving the query metadata from the cache while executing the query plan.
22. The computer program of claim 18 , further comprising instructions for transmitting a request for the additional metadata from the worker node to the master node.
23. The computer program of claim 22 , further comprising instructions for transmitting the additional metadata to the worker node.
24. A system for executing queries in a parallel processing database, comprising a non-transitory computer readable medium and a processor configured to:
receive a query at a master node, the master node comprising a database catalog including metadata defining database objects;
in response to receiving the query at the master node, initiating a catalog server session, taking a snapshot of the metadata, and associating the snapshot of the metadata with the catalog server session, wherein a separate catalog server session is initiated and a separate snapshot of the metadata is taken for each query;
transmit a query plan and query metadata to a worker node, wherein the query plan is based on the query, wherein the query metadata includes metadata for executing the query plan, wherein the query metadata includes database table definitions that define database objects, and wherein the query metadata is retrieved from the snapshot of the metadata associated with the catalog server session;
receive, by the master node, a request for additional metadata that is required for the worker node to execute the query plan, wherein in response to a determination that the worker node requires the additional metadata for the worker node to execute the query plan, the worker node queries a parent in a tree structure of a plurality of worker nodes in the parallel processing database system for the additional metadata, wherein the parent node is a node between the master node and the worker node in relation to the tree structure;
communicate, to the worker node, the additional metadata that is required for the worker node to execute the query plan, wherein the additional metadata is retrieved from a same session as the catalog server session corresponding to the query;
execute the query plan on the worker node; and
return, to the master node, a result associated with the execution of the query plan on the worker node.
25. The system of claim 24 , wherein the processor is further configured to generate a plurality of query plans based on the query.
26. The system of claim 25 , wherein the processor is further configured to transmit the plurality query plans to a plurality of worker nodes.
27. The system of claim 26 , wherein the processor is further configured to execute the plurality of query plans in parallel.
28. The system of claim 24 , wherein the processor is further configured to store the query metadata in a cache on the worker node.