Query plan management in shared distributed data stores
View Patent ↗The invention identifies and caches query plans in a shared-nothing distributed data store that are unlikely to become invalid because they do not reference objects that are likely to be changed or deleted. Plans that are likely to become invalid and are not cached are those plans that reference data that is partitioned across segment/query execution nodes of the data store, plans that are complex, and plans that reference objects that are not “built-in” (primitive) objects. The effect is that most plans which are generated on a query dispatch (master) node are not cached, whereas most plans generated on an execution (segment) node are cached.
1. A method, comprising:
obtaining a query plan by a data store system having a master node and a plurality of segment nodes, each node of the data store system comprising a respective processor and a respective non-transitory storage medium, wherein the master node is a query distributing node, and each segment node is a query executing node;
determining, by the data store system, a likelihood of the query plan becoming invalid, comprising:
determining whether the query plan was generated on the master node or on a segment node of the data store system;
upon determining that the query plan was generated on a segment node, designating the likelihood as low; and
upon determining that the query plan was generated on the master node, performing actions comprising:
determining an estimated complexity value representing an order of complexity of the query plan and designating the query plan as complex or not complex based on the complexity value, wherein more objects being referenced by the query plan correspond to a higher complexity value;
upon designating the query plan as complex, designating the likelihood as high; and
upon designating the query plan as not complex, performing actions comprising:
determining whether the query plan contains one or more objects that are different from built-in objects; and
upon determining that the query plan contains one or more objects that are different from built-in objects, designating the likelihood as low, otherwise designating the likelihood as high; and
caching the query plan upon determining that the likelihood of the query plan becoming invalid is low.
2. The method of claim 1 , wherein said determining the complexity value for the query plan is based on a tree-type data structure of the query plan, the tree-type data structure comprising levels and leaves, wherein more levels and leaves correspond to a higher order of complexity value.
3. The method of claim 1 , wherein designating the query plan as complex occurs when the complexity value is higher than a preselected, user-configurable number.
4. The method of claim 1 , wherein said built-in objects comprise objects registered in the system upon initialization and objects that cannot be removed or altered without causing an undefined state of the system.
5. The method of claim 1 , wherein said caching comprises storing the query plan in the storage medium of the master node or in the storage medium of a segment node of the data store system.
6. The method of claim 1 , wherein said query plan has a tree-type data structure where objects occupy leaf positions in said tree-type data structure, and said determining whether the query plan contains one or more objects that are different from built-in objects comprises searching for objects that are not built-in beginning at said leaf positions.
7. The method of claim 1 further comprising excluding query plans that contain references to data partitioned across said segment nodes from the caching.
8. The method of claim 1 , wherein said caching comprises caching a query plan generated by the master node and concerning metadata.
9. Computer readable non-transitory storage medium product storing executable instructions for causing one or more computers to perform operations comprising:
obtaining a query plan by a data store system having a master node and a plurality of segment nodes, each node of the data store system comprising a respective processor and a respective non-transitory storage medium, wherein the master node is a query distributing node, and each segment node is a query executing node;
determining, by the data store system, a likelihood of the query plan becoming invalid, comprising:
determining whether the query plan was generated on the master node or on a segment node of the data store system;
upon determining that the query plan was generated on the segment node, designating the likelihood of the query plan becoming invalid as low; and
upon determining that the query plan was generated on the master node, performing actions comprising:
determining an estimated complexity value representing complexity of the query plan and designating the query plan as complex or not complex based on the complexity value, wherein more objects being referenced by the query plan correspond to a higher complexity value;
upon designating the query plan as complex, designating the likelihood as high; and
upon designating the query plan as not complex, performing actions comprising:
determining whether the query plan contains one or more objects that are different from built-in objects; and
upon determining that the query plan contains one or more that are different from built-in objects, designating the likelihood of the query plan becoming invalid as low, otherwise designating the likelihood of the query plan becoming invalid as high; and
caching the query plan upon determining that the likelihood of the query plan becoming invalid is low.
10. The computer readable product of claim 9 , wherein designating the query plan as complex occurs when the complexity value is higher than a preselected, user-configurable number.
11. The computer readable product of claim 10 , wherein said plan has a tree-type data structure where objects occupy leaf positions in said tree-type data structure, and said determining whether the query plan contains one or more objects that are different from built-in objects comprises searching for objects that are not built-in beginning at said leaf positions.
12. The computer readable product of claim 9 , wherein said caching comprises caching a query plan generated by the master node and concerning metadata.
13. A data store system, comprising:
a master node comprising a processor;
a plurality of segment nodes each comprising a respective processor, each node of the data store system comprising a respective processor and a respective non-transitory storage medium, wherein the master node is a query distributing node, and each segment node is a query executing node; and
a non-transitory storage medium storing instructions operable to cause the processors to perform operations comprising:
obtaining a query plan by the data store system;
determining, by the data store system, a likelihood of the query plan becoming invalid, comprising:
determining whether the query plan was generated on the master node or on a segment node of the data store system;
upon determining that the query plan was generated on a segment node, designating the likelihood as low; and
upon determining that the query plan was generated on the master node, perform action comprising:
determining an estimated complexity value representing complexity of the query plan and designating the query plan as complex or not complex based on the complexity value, wherein more objects being referenced by the query plan correspond to a higher complexity value;
upon designating the query plan as complex, designating the likelihood as high; and
upon designating the query plan as not complex, performing actions comprising:
determining whether the query plan contains one or more objects that are different from built-in objects; and
upon determining that the query plan contains one or more that are different from built-in objects, designating the likelihood of the query plan becoming invalid as low, otherwise designating the likelihood of the query plan becoming invalid as high; and
caching the query plan upon determining that the likelihood of the query plan becoming invalid is low.
14. The system of claim 13 , wherein said determining the complexity value is based on a tree-type data structure of the query plan, the tree-type data structure comprising levels and leaves, wherein more levels and leaves correspond to a higher complexity.
15. The system of claim 13 , wherein designating the query plan as complex occurs when the complexity is higher than a preselected, user-configurable number.