Methods and systems for load balancing databases in a cloud environment
Systems and methods are presented for load balancing databases in a cloud server environment. In some embodiments, a method can include accessing, by a server in a network-based system, one or more system configuration parameters of the network-based system, with the one or more system configuration parameters defining one or more performance capabilities of the network-based system. The method may also include accessing performance characteristics of a query of a queried database; generating a quadtree decomposition, with the quadtree decomposition modeling a cost estimate of the database query as a function of a range of the performance capabilities of the one or more system configuration parameters; and generating a proposed packing of databases based on the modeled cost estimate of the query including the queried database and defining a configuration of a plurality of databases to be stored in the server.
1. A method comprising:
accessing, by a server in a network-based system, performance characteristics of each query of a query mix of a database;
generating, by the server, a query quadtree decomposition for each query of the query mix, the quadtree decomposition for said query being based on the performance characteristics of said query, the quadtree decomposition for said query modeling a cost estimate of said query as a function of one or more system configuration parameters;
generating, by the server, a mix quadtree decomposition for the query mix based on the quadtree decompositions for each query of the query mix, the mix quadtree decomposition modeling a multi-set of cost estimates of the set of queries when executed concurrently as a function of the one or more system configuration parameters; and
generating a cost quadtree decomposition based on a cost classification function and the mix quadtree decomposition, the cost quadtree decomposition modeling a distribution of a cost-class label for the query mix as a function of the one or more system configuration parameters.
2. The method of claim 1 , wherein:
the query mix is one of a set of query mixes;
the cost quadtree decomposition is one of a set of cost quadtrees, each cost quadtree of the set of cost quadtrees corresponding to a query mix of the set of query mixes; and
the method further comprises:
accessing a recorded frequency distribution of the set of query mixes for a database; and
generating a database quadtree decomposition based on the recorded frequency distribution and the set of mix quadtrees, the database quadtree decomposition modeling a distribution of cost-class label for the database as a function of the one or more system configuration parameters.
3. The method of claim 2 , wherein:
the database is one of a set of databases;
the database quadtree decomposition is one of a set of database quadtree decompositions, each database quadtree of the set of database quadtrees corresponding to a database of the set of databases; and
the method further comprises:
generating, by the server, a packing quadtree decomposition based on the set of database quadtrees, the packing quadtree decomposition modeling a multiset of cost-class label for the set of databases as a function of the performance capabilities of the one or more system configuration parameters to a multiset of class labels, the multiset representing the estimated labels for the databases in the set of databases.
4. The method of claim 3 , further comprising:
generating, by the server, a host quadtree decomposition based on a probability distribution of the system configuration parameters for the host, the host being a server capable of running one or more databases of the set of databases.
5. The method of claim 4 , further comprising:
determining, based on the host quadtree decomposition and the packing quadtree decomposition, a fitness level of running the set of databases on the host.
6. The method of claim 5 , further comprising: comparing the fitness level to
one or more service level objectives; and
based on the fitness level not satisfying the one or more service level objectives, not assigning the set of databases to the host.
7. The method of claim 5 , further comprising:
comparing the fitness level to one or more service level objectives; and
based on the fitness level satisfying the one or more service level objectives, assigning the set of databases to the host.
8. The method of claim 1 , further comprising:
normalizing the one or more system configuration parameters to a unit hypercube with dimension equal to a count of the one or more system configuration parameters; and wherein
the generating of the mix quadtree decomposition is based further on the normalization of the one or more system configuration parameters.
9. The method of claim 1 , wherein the one or more system configuration parameters includes query memory, available buffer pool size, query multi-programming level, prefetch length, buffer pool contents, central processing unit (CPU) cost of operations, or disk transfer time.
10. A system comprising:
a database; and
an optimization module configured to:
access performance characteristics of each query of a query mix of the database;
generate a query quadtree decomposition for each query of the query mix, the quadtree decomposition for said query being based on the performance characteristics of said query, the quadtree decomposition for said query modeling a cost estimate of said query as a function of one or more system configuration parameters;
generate a mix quadtree decomposition for the query mix based on the quadtree decompositions for each query of the query mix, the mix quadtree decomposition modeling a multi-set of cost estimates of the set of queries when executed concurrently as a function of the one or more system configuration parameters; and
generate a cost quadtree decomposition based on a cost classification function and the mix quadtree decomposition, the cost quadtree decomposition modeling a distribution of a cost-class label for the query mix as a function of the one or more system configuration parameters.
11. The system of claim 10 , wherein:
the query mix is one of a set of query mixes;
the cost quadtree decomposition is one of a set of cost quadtrees, each cost quadtree of the set of cost quadtrees corresponding to a query mix of the set of query mixes; and
the optimization module is further configured to:
access a recorded frequency distribution of the set of query mixes for a database; and
generate a database quadtree decomposition based on the recorded frequency distribution and the set of mix quadtrees, the database quadtree decomposition modeling a distribution of cost-class label for the database as a function of the one or more system configuration parameters.
12. The system of claim 11 , wherein:
the database is one of a set of databases;
the database quadtree decomposition is one of a set of database quadtree decompositions, each database quadtree of the set of database quadtrees corresponding to a database of the set of databases; and
the optimization module is further configured to:
generate a packing quadtree decomposition based on the set of database quadtrees, the packing quadtree decomposition modeling a multiset of cost-class label for the set of databases as a function of the performance capabilities of the one or more system configuration parameters to a multiset of class labels, the multiset representing the estimated labels for the databases in the set of databases.
13. The system of claim 12 , wherein the optimization module is further configured to:
generate a host quadtree decomposition based on a probability distribution of the system configuration parameters for the host, the host being a server capable of running one or more databases of the set of databases.
14. The system of claim 13 , wherein the optimization module is further configured to:
determine, based on the host quadtree decomposition and the packing quadtree decomposition, a fitness level of running the set of databases on the host.
15. The system of claim 14 , wherein the optimization module is further configured to:
compare the fitness level to one or more service level objectives; and
based on the fitness level not satisfying the one or more service level objectives, not assign the set of databases to the host.
16. The system of claim 14 , wherein the optimization module is further configured to:
compare the fitness level to one or more service level objectives; and based on the fitness level satisfying the one or more service level objectives, assign the set of databases to the host.
17. The system of claim 10 , wherein the optimization module is further configured to:
normalize the one or more system configuration parameters to a unit hypercube with dimension equal to a count of the one or more system configuration parameters; and wherein
the generating of the mix quadtree decomposition is based further on the normalization of the one or more system configuration parameters.
18. The system of claim 10 , wherein the one or more system configuration parameters includes query memory, available buffer pool size, query multi-programming level, prefetch length, buffer pool contents, central processing unit (CPU) cost of operations, or disk transfer time.