Query plan management in shared distributed data stores
The invention identifies and caches query plans in a shared-nothing distributed data store that are unlikely to become invalid because they do not reference objects that are likely to be changed or deleted. Plans that are likely to become invalid and are not cached are those plans that reference data that is partitioned across segment/query execution nodes of the data store, plans that are complex, and plans that reference objects that are not “built-in” (primitive) objects. The effect is that most plans which are generated on a query dispatch (master) node are not cached, whereas most plans generated on an execution (segment) node are cached.
1. A method, comprising:
obtaining a generated query plan by a shared-nothing data store system having a query distribution master node and a plurality of query execution segment nodes, each query execution segment node of the data store system comprising a respective processor and a respective non-transitory storage medium, the query plan having a tree-type data structure comprising data objects occupying leaf positions organized in hierarchical levels;
determining, by the data store system, whether to cache the query plan based on a likelihood of the query plan becoming invalid, the likelihood of the query plan becoming invalid being determined based on complexity of the query plan, the likelihood being determined using actions comprising:
determining whether a number of leaves in the tree-type data structure is greater than a preselected and user configurable threshold number;
upon determining that the number of leaves is greater than the preselected and user configurable threshold number, designating the likelihood of the query plan becoming invalid as high; and
upon determining that the number of leaves is not greater than the threshold number, performing actions comprising:
determining whether the query plan contains one or more objects that are different from built-in objects represented by metadata; and
upon determining that the query plan contains one or more objects that are different from built-in objects, designating the likelihood of the query plan becoming invalid as low, otherwise designating the likelihood of the query plan becoming invalid as high; and
caching the query plan upon determining that the likelihood of the query plan becoming invalid is low.
2. The method of claim 1 , wherein the built-in objects comprise objects registered in the system upon initialization and objects that cannot be removed or altered without causing an undefined state of the system.
3. The method of claim 1 , wherein determining whether the query plan contains one or more objects that are different from built-in objects comprises searching for objects that are not builtin beginning at a leaf position.
4. The method of claim/further comprising:
determining whether the query plan contains references to data partitioned across the query execution segment nodes; and
upon determining that the query plan contains references to data partitioned across the query execution segment nodes, designating the likelihood of the query plan becoming invalid as not low.
5. The method of claim 1 , further comprising:
determining whether the query plan is a plan concerning metadata; and
upon determining that the query plan is a plan concerning metadata, caching the query plan.
6. At least one computer-readable non-transitory storage medium storing executable instructions for causing one or more computers to perform operations comprising:
obtaining a generated query plan by a shared-nothing data store system having a query distribution master node and a plurality of query execution segment nodes, each query execution segment node of the data store system comprising a respective processor and a respective non-transitory storage medium, the query plan having a tree-type data structure comprising data objects occupying leaf positions organized in hierarchical levels;
determining, by the data store system, whether to cache the query plan based on a likelihood of the query plan becoming invalid, the likelihood of the query plan becoming invalid being determined based on complexity of the query plan, the likelihood being determined using actions comprising:
determining whether a number of leaves in the treetype data structure is greater than a preselected and user configurable threshold number;
upon determining that the number of leaves is greater than the preselected and user configurable threshold number, designating the likelihood of the query plan becoming invalid as high; and
upon determining that the number of leaves is not greater than the threshold number, performing actions comprising:
determining whether the query plan contains one or more objects that are different from built-in objects represented by metadata; and
upon determining that the query plan contains one or more objects that are different from built-in objects, designating the likelihood of the query plan becoming invalid as low, otherwise designating the likelihood of the query plan becoming invalid as high; and
caching the query plan upon determining that the likelihood of the query plan becoming invalid is low.
7. The non-transitory storage medium of claim 6 , wherein the built-in objects comprise objects registered in the system upon initialization and objects that cannot be removed or altered without causing an undefined state of the system.
8. The non-transitory storage medium of claim 6 , wherein determining whether the query plan contains one or more objects that are different from built-in objects comprises searching for objects that are not built-in beginning at a leaf position.
9. The non-transitory storage medium of claim 6 , the operations further comprising:
determining whether the query plan contains references to data partitioned across the query execution segment nodes; and
upon determining that the query plan contains references to data partitioned across the query execution segment nodes, designating the likelihood of the query plan becoming invalid as not low.
10. The non-transitory storage medium of claim 6 , wherein the operations further comprise:
determining whether the query plan is a plan concerning metadata; and
upon determining that the query plan is a plan concerning metadata, caching the query plan.
11. A data store system, comprising:
a query distribution master node comprising a processor;
a plurality of query execution segment nodes each comprising a respective processor and a respective non-transitory storage medium; and
a non-transitory storage medium storing instructions operable to cause the processors to perform operations comprising:
obtaining a generated query plan by the data store system, the query plan having a tree-type data structure comprising data objects occupying leaf positions organized in hierarchical levels;
determining, by the data store system, whether to cache the query plan based on a likelihood of the query plan becoming invalid, the likelihood of the query plan becoming invalid being determined based on complexity of the query plan, the likelihood being determined using actions comprising:
determining whether a number of leaves in the treetype data structure is greater than a preselected and user configurable threshold number;
upon determining that the number of leaves is greater than the preselected and user configurable threshold number, designating the likelihood of the query plan becoming invalid as high; and
upon determining that the number of leaves is not greater than the threshold number, performing actions comprising:
determining whether the query plan contains one or more objects that are different from built-in objects represented by metadata; and
upon determining that the query plan contains one or more objects that are different from built-in objects, designating the likelihood of the query plan becoming invalid as low, otherwise designating the likelihood of the query plan becoming invalid as high; and
caching the query plan upon determining that the likelihood of the query plan becoming invalid is low, wherein the data store system is a shared-nothing data store system.
12. The data store system of claim 11 , wherein the built-in objects comprise objects registered in the system upon initialization and objects that cannot be removed or altered without causing an undefined state of the system.
13. The data store system of claim 11 , wherein determining whether the query plan contains one or more objects that are different from built-in objects comprises searching for objects that are not built-in beginning at a leaf position.
14. The data store system of claim 11 , the operations further comprising:
determining whether the query plan contains references to data partitioned across the query execution segment nodes; and
upon determining that the query plan contains references to data partitioned across the query execution segment nodes, designating the likelihood of the query plan becoming invalid as not low.
15. The data store system of claim 11 , wherein the operations further comprise:
determining whether the query plan is a plan concerning metadata; and
upon determining that the query plan is a plan concerning metadata, caching the query plan.