Parallel processing database system with a shared metadata store
A method and system for executing database queries in parallel using a shared metadata store. The metadata store may reside on a master node, and the master node may distribute this metadata with query plans to worker nodes. The worker nodes may request additional metadata from the master node when necessary.
1. A parallel processing database system, comprising:
a master node comprising one or more processors, wherein the master node is configured to receive a query from a client node, the query being communicated to the master node form the client node via one or more networks, develop a query plan based at least in part on the query, determine metadata to be used in connection with execution of the query plan, and to communicate, to a first worker node via at least one of the one or more networks, the query plan and the metadata determined to be used in connection with execution of the query plan, wherein the metadata is stored in a database catalog that comprises database objects, and wherein the metadata is associated with corresponding ones of the database objects;
the first worker node comprising one or more processors, wherein the first worker node is in communication with the master node and receives the query plan from the master node, executes at least a part of the query in connection with executing the query plan received from the master node, and in an event that the first worker node requires additional metadata in connection with executing the at least the part of the query, communicates a request for the additional metadata to a parent node corresponding to the first worker node; and
the parent node comprising one or more processors, wherein in the event that the parent node does not have the additional metadata, the parent node forwards the request for the additional metadata to a next parent node in a hierarchical structure of nodes.
2. The system of claim 1 , wherein the parent node corresponding to the first worker node is determined in relation to the hierarchical structure of nodes.
3. The system of claim 2 , wherein hierarchical structure of nodes includes the master node, the first worker node, and the parent node arranged in a hierarchy.
4. The system of claim 1 , wherein in the event that the parent node has the additional metadata, the parent node communicates the additional metadata to the first worker node.
5. The system of claim 1 , wherein the first worker node comprises a query execution process that is configured to execute a database query, wherein the first worker node runs the query execution process in response to receiving the query plan from the master node.
6. The system of claim 5 , wherein the database query comprises the query plan and query metadata.
7. The system of claim 6 , wherein the query metadata is stored in the database catalog and corresponds to the metadata that is associated with the database objects and that is determined to be used in connection with execution of the query plan.
8. The system of claim 1 , wherein the master node comprises a query dispatch process.
9. The system of claim 8 , wherein the query dispatch process receives the query from the client node.
10. The system of claim 9 , wherein the query dispatch process creates the query plan based at least in part on the query.
11. The system of claim 10 , wherein the query dispatch process transmits the query plan to the first worker node.
12. The system of claim 8 , wherein the query dispatch process obtains query metadata from the database catalog, wherein the query metadata corresponds to the metadata that is associated with the database objects and that is determined to be used in connection with execution of the query plan.
13. The system of claim 12 , wherein the query dispatch process transmits the query metadata to the first worker node.
14. The system of claim 8 , wherein the query dispatch process transmits a query plan and query metadata concurrently to the first worker node.
15. The system of claim 1 , further comprising a plurality of worker nodes, wherein the first worker node corresponds to at least one of the plurality of worker nodes.
16. The system of claim 15 , wherein the parent node corresponds to at least one of the plurality of worker nodes.
17. The system of claim 15 , wherein the master node communicates a plurality of query plans and corresponding query metadata to the plurality of worker nodes.
18. The system of claim 17 , wherein the plurality of worker nodes execute the query plans in parallel.
19. The system of claim 1 , wherein the database objects comprise one or more of database table definitions, database functions, database users, database views, and database indexes.
20. The system of claim 1 , wherein the first worker node executes the query plan using at least the metadata received from the master node.
21. The system of claim 1 , wherein the parent node is another worker node connected to the master node via the hierarchical structure of nodes.
22. The system of claim 1 , wherein the first worker node stores its ancestry in relation to the hierarchical structure of nodes.
23. The system of claim 1 , wherein in an event that a change is made to the metadata in the database catalog, the change is not propagated to the worker nodes to update a database at the worker nodes.
24. A method for processing a query, comprising:
receiving, by a worker node, a query plan and metadata from a master node via at least one of one or more networks, wherein the query plan is generated by the master node based at least in part on a query that the master node receives from a client node, the query being communicated to the master node form the client node via one or more networks, wherein the metadata is determine metadata to be used in connection with execution of the query plan, and wherein the metadata stored in a database catalog that comprises database objects, and wherein the metadata is associated with corresponding ones of the database objects;
executing, by the worker node, at least a part of the query in connection with executing the query plan received from the master node; and
in an event that the worker node requires additional metadata in connection with executing the at least the part of the query, communicating a request for the additional metadata to a parent node corresponding to the worker node, wherein in the event that the parent node does not have the additional metadata, the parent node forwards the request for the additional metadata to a next parent node in a hierarchical structure of nodes.