Query plans for analytic SQL constructs
A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.
1. A method comprising:
initiating, within a first execution node of an execution platform, a first operator in a query plan to process a set of data and generate an intermediate result of a query;
determining whether the first operator has finished processing the set of data; and
after determining that the first operator has finished processing the set of data:
pushing, during execution of the query plan, the intermediate result of the first operator to a plurality of secondary operators as the intermediate result is generated;
initiating, by one or more processors, each of the other secondary operators to concurrently process the intermediate result to generate a plurality of second results, wherein a timing of processing of the intermediate result by each of the other secondary operators is adjusted to coordinate the generation of the plurality of second results based on a final operation to be executed on the plurality of second results;
executing the final operation on the plurality of second results to generate a final result; and
storing the final result to disk storage.
2. The method of claim 1 , wherein the final result is generated without storing the intermediate result to the disk storage.
3. The method of claim 1 , wherein each of the plurality of secondary operators process the intermediate result with a different operation.
4. The method of claim 3 , further comprising delaying operation of at least one of the plurality of secondary operators.
5. The method of claim 4 , wherein operation of at least one of the plurality of secondary operators is delayed to coordinate the generation of the plurality of second results.
6. The method of claim 1 , wherein the intermediate result generated by the first operator is not recomputed by any of the plurality of secondary operators, and wherein the intermediate result is processed by the plurality of secondary operators to execute a plurality of different queries.
7. The method of claim 1 , wherein each of the plurality of secondary operators are unique operators.
8. The method of claim 1 , wherein the intermediate result is not materialized.
9. The method of claim 1 , wherein at least one of the plurality of secondary operators processes the intermediate result during execution of the query plan.
10. A system comprising:
a memory; and
one or more processors, operatively coupled to the memory, the one or more processors to:
initiate, within a first execution node of an execution platform, a first operator in a query plan to process a set of data and generate an intermediate result of a query;
determine whether the first operator has finished processing the set of data; and
after determining that the first operator has finished processing the set of data:
push, during execution of the query plan, the intermediate result of the first operator to a plurality of secondary operators as the intermediate result is generated;
initiate each of the other secondary operators to concurrently process the intermediate result to generate a plurality of second results, wherein a timing of processing of the intermediate result by each of the other secondary operators is adjusted to coordinate the generation of the plurality of second results based on a final operation to be executed on the plurality of second results;
execute the final operation on the plurality of second results to generate a final result; and
store the final result to disk storage.
11. The system of claim 10 , wherein the one or more processors generate the final result without storing the intermediate result to the disk storage.
12. The system of claim 10 , wherein each of the plurality of secondary operators process the intermediate result with a different operation.
13. The system of claim 12 , wherein the one or more processors are further to delay operation of at least one of the plurality of secondary operators.
14. The system of claim 13 , wherein the one or more processors delay operation of at least one of the plurality of secondary operators to coordinate the generation of the plurality of second results.
15. The system of claim 10 , wherein the intermediate result generated by the first operator is not recomputed by any of the plurality of secondary operators, and wherein the intermediate result is processed by the plurality of secondary operators to execute a plurality of different queries.
16. The system of claim 10 , wherein each of the plurality of secondary operators are unique operators.
17. The system of claim 10 , wherein the one or more processors do not materialize the intermediate result.
18. The system of claim 10 , wherein at least one of the plurality of secondary operators processes the intermediate result during execution of a query plan.
19. A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors cause the one or more processors to:
initiate, within a first execution node of an execution platform, a first operator in a query plan to process a set of data and generate an intermediate result of a query;
determine whether the first operator has finished processing the set of data; and
after determining that the first operator has finished processing the set of data:
push, during execution of the query plan, the intermediate result of the first operator to a plurality of secondary operators as the intermediate result is generated;
initiate, by the one or more processors, each of the other secondary operators to concurrently process the intermediate result to generate a plurality of second results, wherein a timing of processing of the intermediate result by each of the other secondary operators is adjusted to coordinate the generation of the plurality of second results based on a final operation to be executed on the plurality of second results;
operate on the plurality of second results to generate a final result; and
store the final result to disk storage.
20. The non-transitory computer-readable medium of claim 19 , wherein the one or more processors generate the final result without storing the intermediate result to the disk storage.
21. The non-transitory computer-readable medium of claim 19 , wherein each of the plurality of secondary operators process the intermediate result with a different operation.
22. The non-transitory computer-readable medium of claim 19 , wherein the one or more processors are further to delay operation of at least one of the plurality of secondary operators.
23. The non-transitory computer-readable medium of claim 22 , wherein the one or more processors delay operation of at least one of the plurality of secondary operators to coordinate the generation of the plurality of second results.
24. The non-transitory computer-readable medium of claim 23 , wherein the intermediate result generated by the first operator is not recomputed by any of the plurality of secondary operators, and wherein the intermediate result is processed by the plurality of secondary operators to execute a plurality of different queries.
25. The non-transitory computer-readable medium of claim 19 , wherein each of the plurality of secondary operators are unique operators.
26. The non-transitory computer-readable medium of claim 19 , wherein the one or more processors do not materialize the intermediate result.
27. The non-transitory computer-readable medium of claim 19 , wherein at least one of the plurality of secondary operators processes the intermediate result during execution of a query plan.