Data-driven query-execution scheduling
Embodiments of the present disclosure may provide a database optimization system that can generate computational values through a database compiler and assignment data for execution of a query by a plurality of nodes of a database system. The computational values and assignment data can be generated by one or more machine learning schemes. The machine learning schemes can be trained on previous computational values and previous assignment data.
1. A method comprising:
receiving a query for execution on a dataset of a distributed database using a plurality of nodes;
generating, by a database compiler of the distributed database, a query graph for execution of the query and database metadata of the dataset;
generating a predicted computational resource profile by applying a first machine learning scheme on the query graph and the database metadata, the predicted computational resource profile comprising predicted computational usage values for execution of the query using the plurality of nodes, the first machine learning scheme trained on a historical database dataset comprising previous query plans of previous queries and previous predicted resource profiles implemented for the previous query plans;
generating assignment data to execute processes of the query on the plurality of nodes, the assignment data generated by applying a second machine learning scheme to current state data of a current computational state of the plurality of nodes and the predicted computational resource profile generated by the first machine learning scheme, the assignment data assigning nodes and corresponding computational node usage values to the plurality of nodes for execution the query;
executing the query on the plurality of nodes according to the assignment data; and
storing query results.
2. The method of claim 1 , wherein the predicted computational resource profile comprises a vector of computational resource values for use in query execution on the plurality of nodes.
3. The method of claim 2 , wherein the computational resource values comprise computer processor usage values, memory values, network activity values of the plurality of nodes.
4. The method of claim 1 , further comprising:
requesting, over a network, current state data from the plurality of nodes; and
receiving, other the network, the current state data from the plurality of nodes.
5. The method of claim 4 , wherein the current state data comprises computational resource values of each node in the plurality of nodes.
6. The method of claim 1 , wherein the historical database dataset further comprises previous state data sets of the plurality of nodes and corresponding previous assignment data sets implemented for corresponding previous state data items.
7. The method of claim 6 , wherein each previous assignment data set is assignment data that was implemented for a particular previous state data set of the previous state data sets.
8. The method of claim 6 , wherein the second machine learning scheme is trained on the previous predicted computational resource profiles and the corresponding previous state data sets of the plurality of nodes.
9. The method of claim 1 , wherein the query is received from an end-user of a client device.
10. The method of claim 9 , wherein the historical database dataset further comprises user usage data describing previous query workloads received from the end-user for execution on the distributed database.
11. The method of claim 10 , wherein the second machine learning scheme is trained to generate the assignment data based on the user usage data, the previous predicted computational resource profiles and previous state data sets of the plurality of nodes that correspond to the previous predicted computational resource profiles.
12. The method of claim 1 , wherein a plurality of additional queries are queued for execution with the query in a query pool.
13. The method of claim 12 , wherein the second machine learning scheme is trained to generate the assignment data based on a quantity of queries in the query pool.
14. The method of claim 13 , wherein the second machine learning scheme is trained to generate the assignment data based on a query type for each of the plurality of additional queries in the query pool.
15. The method of claim 14 , wherein the query type comprises one or more of: a database select operation, a database delete operation, and a database insert operation.
16. The method of claim 1 , wherein the first machine learning scheme is a first decision tree, and wherein the second machine learning scheme is a second decision tree.
17. The method of claim 1 , wherein the first machine learning scheme is a first neural network, and wherein the second machine learning scheme is a second neural network.
18. The method of claim 1 , wherein the query plan is a directed acyclic graph of tasks to execute the query on the distributed database.
19. A system comprising:
one or more processors of a machine; and
at least one memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving a query for execution on a dataset of a distributed database using a plurality of nodes;
generating, by a database compiler of the distributed database, a query graph for execution of the query and database metadata of the dataset;
generating a predicted computational resource profile by applying a first machine learning scheme on the query graph and database metadata, the predicted computational resource profile comprising predicted computational usage values for execution of the query using the plurality of nodes, the first machine learning scheme trained on a historical database dataset comprising previous query plans of previous queries and previous predicted computational resource profiles implemented for the previous query plans;
generating assignment data to execute processes of the query on the plurality of nodes, the assignment data generated by applying a second machine learning scheme to current state data of a current computational state of the plurality of nodes and the predicted computational resource profile generated by the first machine learning scheme, the assignment data assigning nodes and corresponding computational node usage values to the plurality of nodes for execution the query;
executing the query on the plurality of nodes according to the assignment data; and
storing query results.
20. The system of claim 19 , wherein the predicted computational resource profile comprises a vector of computational resource values for use in query execution on the plurality of nodes.
21. The system of claim 20 , wherein the computational resource values comprise computer processor usage values, memory values, network activity values of the plurality of nodes.
22. The system of claim 19 , the operations further comprising:
requesting, over a network, current state data from the plurality of nodes; and receiving, other the network, the current state data from the plurality of nodes.
23. The system of claim 22 , wherein the current state data comprises computational resource values of each node in the plurality of nodes.
24. A machine-storage media embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving a query for execution on a dataset of a distributed database using a plurality of nodes;
generating, by a database compiler of the distributed database, a query graph for execution of the query and database metadata of the dataset;
generating a predicted computational resource profile by applying a first machine learning scheme on the query graph and database metadata, the predicted computational resource profile comprising predicted computational usage values for execution of the query using the plurality of nodes, the first machine learning scheme trained on a historical database dataset comprising previous query plans of previous queries and previous predicted computational resource profiles implemented for the previous query plans;
generating assignment data to execute processes of the query on the plurality of nodes, the assignment data generated by applying a second machine learning scheme to current state data of a current computational state of the plurality of nodes and the predicted computational resource profile generated by the first machine learning scheme, the assignment data assigning nodes and corresponding computational node usage values to the plurality of nodes for execution the query;
executing the query on the plurality of nodes according to the assignment data; and
storing query results.
25. The machine-storage media of claim 24 , wherein the predicted computational resource profile comprises a vector of computational resource values for use in query execution on the plurality of nodes.