Parallel processing database system with a shared metadata store
A method and system for executing database queries in parallel using a shared metadata store. The metadata store may reside on a master node, and the master node may distribute this metadata with query plans to worker nodes. The worker nodes may request additional metadata from the master node when necessary.
1. A parallel processing database system, comprising:
a master node comprising a processor and a database catalog, wherein the database catalog includes metadata defining database objects, and wherein the master node is configured to receive a query from a client node, wherein the master node includes a query dispatcher that receives the query from the client node, develops the query plan based at least in part on the query, and communicates the query plan to a worker node in connection with metadata from the database catalog that the query dispatcher determines is required for execution of the query plan; and
the worker node, wherein the worker node is in communication with the master node such that the worker node receives a query plan from the master node, and the worker node comprising a processor and a query execution process, wherein the processor runs the query execution process so as to execute at least a part of the query in connection with executing the query plan received from the master node, and wherein in an event that the worker node requires additional metadata in connection with executing the at least the part of the query, the processor makes a request, for the additional metadata, to a parent node corresponding to the worker node in relation to a hierarchical structure of nodes,
wherein the hierarchical structure of nodes includes the master node, the worker node, and the parent node arranged in a hierarchy, and
wherein in the event that the parent node does not have the additional metadata, the parent node forwards the request for the additional metadata to the next parent node in the hierarchical structure of nodes.
2. The system of claim 1 , wherein the master node comprises a processor and a non-transitory computer readable storage medium.
3. The system of claim 1 , wherein the worker node comprises a processor and a non-transitory computer readable storage medium.
4. The system of claim 1 , wherein the query execution process executes a database query received from the master node, and wherein the query plan is included in the database query received from the master node.
5. The system of claim 4 , wherein the database query comprises a query plan and query metadata.
6. The system of claim 5 , wherein the query metadata is stored in the database catalog.
7. The system of claim 1 , wherein the master node comprises a query dispatch process.
8. The system of claim 7 , wherein the query dispatch process receives the query from the client.
9. The system of claim 8 , wherein the query dispatch process creates the query plan from the query.
10. The system of claim 9 , wherein the query dispatch process transmits the query plan to the worker node.
11. The system of claim 7 , wherein the query dispatch process retrieves query metadata from the database catalog.
12. The system of claim 11 , wherein the query dispatch process transmits the query metadata to the worker node.
13. The system of claim 7 , wherein the query dispatch process transmits a query plan and query metadata concurrently to the worker node.
14. The system of claim 1 , further comprising a plurality of worker nodes.
15. The system of claim 14 , wherein the master node transmits a plurality of query plans and query metadata to the plurality of worker nodes.
16. The system of claim 15 , wherein the plurality of worker nodes execute the query plans in parallel.
17. The system of claim 1 , wherein the database objects comprise database table definitions, database functions, database users, database views, and database indexes.
18. The system of claim 1 , wherein the worker node comprises a non-transitory computer readable storage medium.
19. The system of claim 1 , wherein the query execution process of the worker node executes the query plan using at least the metadata received from the master node.
20. The system of claim 1 , wherein the parent node is another worker node connected to the master node via the hierarchical structure.
21. The system of claim 1 , wherein the worker node stores its ancestry in relation to the hierarchical structure of nodes.
22. The system of claim 1 , wherein in an event that a change is made to the metadata in the database catalog, the change is not propagated to the worker nodes to update a database at the worker nodes.