Query plans for analytic SQL constructs
A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.
1. A non-transitory computer-readable medium storing instructions which, when executed by one or more third processors of a computing device, cause the computing device to:
process, within a first processor, a set of data with a first operator of a query plan for a query to generate an intermediate result of the query plan;
push, by the first processor, during execution of the query plan, the intermediate result to a plurality of second processors in the query plan for processing by a plurality of parallel operators executing within the plurality of second processors that generate a plurality of second results, wherein each of the plurality of second processors store at least some of the plurality of intermediate results in a local cache that corresponds to that second processor; and
store a final result to a persistent storage device within a storage platform.
2. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the computing device to:
access the query plan for the set of data referenced by the query.
3. The non-transitory computer-readable medium of claim 1 , wherein each of the first processor and the one or more second processors are decoupled from the persistent storage device.
4. The non-transitory computer-readable medium of claim 1 , wherein the plurality of second results is generated without writing the intermediate result to the persistent storage device.
5. The non-transitory computer-readable medium of claim 1 , wherein each of the plurality of parallel operators processes the intermediate result with a different operation.
6. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the computing device to:
delay operation of at least one of the plurality of parallel operators so as to coordinate timing among other parallel operators of the plurality of parallel operators.
7. The non-transitory computer-readable medium of claim 1 , wherein the intermediate result comprises a plurality of rows of database data.
8. The non-transitory computer-readable medium of claim 1 , wherein each of the plurality of parallel operators are unique operators.
9. The non-transitory computer-readable medium of claim 1 , wherein the intermediate result is not materialized.
10. The non-transitory computer-readable medium of claim 1 , wherein the intermediate result generated by the first operators is not materialized to a temporary structure.
11. The non-transitory computer-readable medium of claim 1 , wherein the each of the first processor and second plurality of processors is coupled to the persistent storage device via a communications network.
12. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the computing device to:
receive the query for information stored in one or more databases.
13. A method comprising:
processing, within a first processor, a set of data with a first operator of a query plan for a query to generate an intermediate result of the query plan;
pushing, by the first processors, during execution of the query plan, the intermediate result to plurality of second processors in the query plan for processing by a plurality of parallel operators executed within the plurality of second processors that generate a plurality of second results, wherein each of the one or more second processors store at least some of the plurality of intermediate results in a local cache that corresponds to that second processor; and
storing a final result to a persistent storage device within a storage platform.
14. The method of claim 13 , further comprising:
accessing the query plan for the set of data references by the query.
15. The method of claim 13 , wherein each of the first processor and the one or more second processors are decoupled from the persistent storage device.
16. The method of claim 13 , wherein the plurality of second results generated without writing the intermediate result to the persistent storage device.
17. The method of claim 13 , wherein each of the plurality of parallel operators process the intermediate result with a different operation.
18. The method of claim 13 , further comprising:
delaying operation of at least one of the plurality of parallel operators so as to coordinate timing among other parallel operators of the plurality of parallel operators.
19. The method of claim 13 , wherein the intermediate result comprises a plurality of rows of database data.
20. The method of claim 13 , wherein each of the plurality of parallel operators are unique operators.
21. The method of claim 13 , wherein the intermediate result is not materialized.
22. The method of claim 13 , wherein the intermediate result generated by the first operators is not materialized to a temporary structure.
23. The method of claim 13 , wherein the each of the first processor and the second plurality of processors is coupled to the persistent storage device via a communications network.
24. The method of claim 13 , wherein the instructions further cause the computing device to:
receiving the query for information stored in one or more databases.
25. A system comprising:
a first processor programmed to,
process a set of data with a first operator of a query plan for a query to generate an intermediate result of the query plan, and
push, during execution of the query plan, the intermediate result to a plurality of second processors in the query plan for concurrent processing by the plurality of parallel operators;
the plurality of second processors programmed to generate the plurality of second results, wherein each of the plurality of second processors store at least some of the plurality of intermediate results in a local cache that corresponds to that second processor; and
a storage platform to store a final result to a persistent storage device.
26. The system of claim 25 , wherein each of the first processor and the one or more second processors are decoupled from the persistent storage device.
27. The system of claim 25 , wherein the plurality of second results generated without writing the intermediate result to the persistent storage device.
28. The system of claim 25 , wherein each of the plurality of parallel operators process the intermediate result with a different operation.