Metadata versioning in a distributed database
In one respect, there is provided a distributed database system. The distributed database system can include a plurality of nodes and a plurality of optimizers. A first optimizer of the plurality of optimizers can be configured to: receive, from a client, a command; generate, based at least in part on global metadata cached by the first optimizer, an execution plan for fulfilling the command that includes one or more tasks to be performed by at least a first node of the plurality of nodes; determine that at least one task of the one or more tasks are not successfully performed by the first node; and in response to determining that at least one task of the one or more tasks are not successfully performed by the first node, updating the cached global metadata at the first optimizer.
1. A system, comprising:
at least one data processor; and
at least one memory storing instructions, which when executed by the at least one data processor, result in operations comprising:
receiving, at a first optimizer, a command on a distributed database system including the first optimizer and a plurality of nodes;
generating, by the first optimizer, an execution plan for fulfilling the command, wherein the execution plan includes a task to be performed by a first node of the plurality of nodes, wherein the execution plan is generated based at least on global metadata cached at the first optimizer, and wherein the execution plan is generated optimistically without the first optimizer verifying that the global metadata cached at the first optimizer as having been synchronized with a global copy of a global metadata associated with the distributed database to include a most recent update to the global metadata associated with the distributed database system;
determining, by the first optimizer, that the first node is unable to successfully perform the task based on the execution plan generated using the global metadata cached at the first optimizer; and
in response to determining that the task is not successfully performed by the first node, updating the global metadata cached at the first optimizer by at least synchronizing the global metadata cached at the first optimizer with the global copy of the global metadata associated with the distributed database system.
2. The system of claim 1 , further comprising:
distributing, to the first node, the task and an indicator corresponding to a version number of the global metadata used to generate the execution plan including the task; and
receiving, from the first node, an indication corresponding to whether the first node is able to successfully perform the task, wherein the indication is generated by the first node based at least on the indicator from the first optimizer and a version number of local metadata at the first node.
3. The system of claim 2 , wherein the first node is unable to successfully perform the task, when the local metadata at the first node does not match the global metadata cached at the first optimizer.
4. The system of claim 3 , wherein the local metadata at the first node does not match the global metadata cached at the first optimizer, when the local metadata at the first node comprises a more recent version of the global metadata than the global metadata cached at the first optimizer.
5. The system of claim 1 , further comprising:
re-executing the command by at least generating, based at least in part on the updated global metadata at the first optimizer, another execution plan for fulfilling the command.
6. The system of claim 1 , wherein the command comprises a data definition language (DDL) command modifying a schema and/or one or more database tables comprising the distributed database, and wherein modifying the schema and/or one or more database tables includes modifying the global metadata associated with the distributed database system.
7. The system of claim 6 , wherein the task includes one or more updates to a local metadata at the first node.
8. The system of claim 6 , further comprising:
creating a lock on the global metadata associated with the distributed database system;
updating the global copy of the global metadata associated with the distributed database system; and
releasing the lock on the global metadata upon updating the global copy of the global metadata associated with the distributed database system.
9. The system of claim 8 ,
wherein a second optimizer in the distributed database system is configured to respond to the release of the lock on the global metadata at least by updating global metadata cached by the second optimizer, and wherein the updating of the global metadata cached by the second optimizer includes synchronizing the global metadata cached at the second optimizer with the global copy of the global metadata.
10. The system of claim 1 , wherein the command comprises a data modification language (DML) command, and wherein the task includes one or more operations on data stored at and/or managed by the first node.
11. A method, comprising:
receiving, at a first optimizer, a command on a distributed database system including the first optimizer and a plurality of nodes;
generating, by the first optimizer, an execution plan for fulfilling the command, wherein the execution plan includes a task to be performed by a first node of the plurality of nodes, wherein the execution plan is generated based at least on global metadata cached at the first optimizer, and wherein the execution plan is generated optimistically without the first optimizer verifying that the global metadata cached at the first optimizer as having been synchronized with a global copy of a global metadata associated with the distributed database to include a most recent update to the global metadata associated with the distributed database system;
determining, by the first optimizer, that the first node is unable to successfully perform the task based on the execution plan generated using the global metadata cached at the first optimizer; and
in response to determining that the task is not successfully performed by the first node, updating the global metadata cached at the first optimizer by at least synchronizing the global metadata cached at the first optimizer with the global copy of the global metadata associated with the distributed database system.
12. The method of claim 11 , further comprising:
distributing, to the first node, the task and an indicator corresponding to a version number of the global metadata used to generate the execution plan including the task; and
receiving, from the first node, an indication corresponding to whether the first node is able to successfully perform the task, wherein the indication is generated by the first node based at least on the indicator and a version number of local metadata at the first node.
13. The method of claim 12 , wherein the first node is unable to successfully perform the task, when the local metadata at the first node does not match the global metadata cached at the first optimizer.
14. The method of claim 11 , further comprising:
re-execute, by the first optimizer, the command by at least generating, based at least in part on the updated global metadata at the first optimizer, another execution plan for fulfilling the command.
15. The method of claim 11 , wherein the command comprises a data definition language (DDL) command modifying a schema and/or one or more database tables comprising the distributed database, and wherein modifying the schema and/or one or more database tables includes modifying the global metadata associated with the distributed database system.
16. The method of claim 15 , further comprising:
creating a lock on the global metadata associated with the distributed database system;
updating the global copy of the global metadata associated with the distributed database system; and
releasing the lock on the global metadata upon updating the global copy of the global metadata associated with the distributed database system.
17. The method 17 , wherein a second optimizer in the distributed database system is configured to respond to the release of the lock on the global metadata at least by updating global metadata cached by the second optimizer, and wherein the updating of the global metadata cached by the second optimizer includes synchronizing the global metadata cached at the second optimizer with the global copy of the global metadata.
18. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
receiving, at a first optimizer, a command on a distributed database system including the first optimizer and a plurality of nodes;
generating, by the first optimizer, an execution plan for fulfilling the command, wherein the execution plan includes a task to be performed by a first node of the plurality of nodes, wherein the execution plan is generated based at least on global metadata cached at the first optimizer, and wherein the execution plan is generated optimistically without the first optimizer verifying that the global metadata cached at the first optimizer as having been synchronized with a global copy of a global metadata associated with the distributed database to include a most recent update to the global metadata associated with the distributed database system;
determining, by the first optimizer, that the first node is unable to successfully perform the task based on the execution plan generated using the global metadata cached at the first optimizer; and
in response to determining that the task is not successfully performed by the first node, updating the global metadata cached at the first optimizer by at least synchronizing the global metadata cached at the first optimizer with the global copy of the global metadata associated with the distributed database system.