SYSTEMS AND METHODS FOR TUNING MULTI-STORE SYSTEMS TO SPEED UP BIG DATA QUERY WORKLOAD
Systems and methods are disclosed to run a multistore system by receiving by-products of query processing in the multistore system, wherein the by-products include views or materializations of intermediate data; placing the views or materializations across the stores based on recently observed queries as indicative of a future query workload; determining a benefit score for each view based on a predicted future query workload, wherein each store has an allotted view storage budget, and there is a view transfer budget for transferring views between the stores; and tuning a physical design of the multistore system.
1 . A method to run a multistore system, comprising:
receiving by-products of query processing in the multistore system, wherein the by-products include views or materializations of intermediate data;
placing the views or materializations across the stores based on recently observed queries as indicative of a future query workload;
determining a benefit score for each view based on a predicted future query workload, wherein each store has an allotted view storage budget, and there is a view transfer budget for transferring views between the stores; and
tuning a physical design of the multistore system.
2 . The method of claim 1 , comprising transferring views between the stores to fit a final view placement within all of budgets and minimize a cost of the future query workload.
3 . The method of claim 1 , wherein each budget comprises a unique value.
4 . The method of claim 1 , comprising considering a unified set of views as a union of all views present in each store.
5 . The method of claim 1 , comprising generating solutions including a subset of views to be placed in each store.
6 . The method of claim 1 , comprising solving a view placement for a high-performance store first.
7 . The method of claim 6 , comprising determining views to be most beneficial for the future workload to transfer to (or retain within) the high performance store.
8 . The method of claim 1 , comprising solution is computed for the high performance store and the solution is the set of views to place in the high performance store.
9 . The method of claim 1 , wherein the solution for the high performance store is below a view storage budget for the high performance store and a view transfer budget.
10 . The method of claim 1 , comprising determining a solution for a second store.
11 . The method of claim 1 , comprising determining a solution for the second store with a cost below a view storage budget for the high performance store and below remaining view transfer budgets not consumed by the solution for the high performance store.
12 . The method of claim 1 , comprising deciding which view should be placed in which store and which views should be discarded.
13 . The method of claim 1 , comprising deciding the most beneficial set of views for a given storage budget.
14 . The method of claim 1 , comprising deciding the most beneficial set of views to transfer across stores.
15 . The method of claim 1 , comprising selecting the most beneficial views for a RDBMS.
16 . The method of claim 1 , comprising determining which views for the big data store is chosen next.
17 . The method of claim 1 , comprising handling interactions between views.
18 . The method of claim 1 , comprising using a dynamic programming solution to pack both stores.
19 . A multistore system, comprising:
a processor; and
computer readable code for:
receiving by-products of query processing in the multistore system, wherein the by-products include views or materializations of intermediate data;
placing the views or materializations across the stores based on recently observed queries as indicative of a future query workload;
determining a benefit score for each view based on a predicted future query workload, wherein each store has an allotted view storage budget, and there is a view transfer budget for transferring views between the stores; and
tuning a physical design of the multistore system.
20 . The system of claim 19 , comprising transferring views between the stores to fit a final view placement within all of budgets and minimize a cost of the future query workload.