Query plans for analytic SQL constructs
A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.
1. A method comprising:
initiating, within a first execution node of an execution platform, a first operator in a query plan to process a set of data and generate an intermediate result of a query;
determining whether the first operator has produced an output from processing the set of data;
responsive to determining that the first operator has generated some output:
pushing, during execution of the query plan, the output of the first operator to a plurality of secondary operators in the query plan for concurrent processing by the plurality of secondary operators; and
initiating each of the other secondary operators to process the intermediate result to generate a plurality of second results;
operating on the intermediate result to generate a final result; and
storing the final result to disk storage within a storage platform that is separate from the execution platform.
2. The method of claim 1 , wherein the first result and the plurality of second results are generated without writing the result data to disk storage.
3. The method of claim 1 , wherein each of the plurality of secondary operators process the intermediate result with a different operation.
4. The method of claim 3 , further comprising delaying operation of at least one of the plurality of secondary operators so as to coordinate timing among other secondary operators of the plurality of secondary operators.
5. The method of claim 1 , wherein the intermediate result generated by the first operator is not recomputed and is processed by the plurality of secondary operators to execute a plurality of different queries.
6. The method of claim 1 , wherein each of the plurality of secondary operators are unique operators.
7. The method of claim 1 , wherein the intermediate result is not materialized.
8. A non-transitory machine readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to:
initiate, within a first execution node of an execution platform, a first operator in a query plan to process a set of data and generate an intermediate result of a query;
determine, by the one or more processors, whether the first operator has produced an output from processing the set of data; and
responsive to determining that the first operator has generated some output:
push, during execution of the query plan, the first operator output to a plurality of secondary operators in the query plan for concurrent processing by the plurality of secondary operators; and
initiate each of the other secondary operators to process the intermediate result to generate a plurality of second results;
operate on the intermediate result to generate a final result; and
store the final result to disk storage within a storage platform that is separate from the execution platform.
9. The non-transitory machine readable medium of claim 8 , wherein the first result and the plurality of second results are generated without writing the result data to disk storage.
10. The non-transitory machine readable medium of claim 8 , wherein each of the plurality of secondary operators process the intermediate result with a different operation.
11. The non-transitory machine readable medium of claim 10 , wherein the one or more processors to delay operation of at least one of the plurality of secondary operators so as to coordinate timing among other secondary operators of the plurality of secondary operators.
12. The non-transitory machine readable medium of claim 8 , wherein the intermediate result generated by the first operator is not recomputed and is processed by the plurality of secondary operators to execute a plurality of different queries.
13. The non-transitory machine readable medium of claim 8 , wherein each of the plurality of secondary operators are unique operators.
14. The non-transitory machine readable medium of claim 8 , wherein the intermediate result is not materialized.
15. A system, comprising:
one or more processors to:
initiate, within a first execution node of an execution platform, a first operator in a query plan to process a set of data and generate an intermediate result of a query without writing result data to the disk storage within a storage platform that is separate from the execution platform;
push, during execution of the query plan, the output of the first operator to a plurality of secondary operators in the query plan for concurrent processing by the plurality of secondary operators to generate a plurality of second results; and
operate on the intermediate result to generate a final result; and
store the final result to disk storage within the storage platform.
16. The system of claim 15 , wherein each of the plurality of secondary operators process the intermediate result with a different operation.
17. The system of claim 16 , further comprising delaying operation of at least one of the plurality of secondary operators so as to coordinate timing among other secondary operators of the plurality of secondary operators.
18. The system of claim 15 , wherein the intermediate result generated by the first operator is not recomputed and is processed by the plurality of secondary operators to execute a plurality of different queries.
19. The system of claim 15 , wherein the intermediate result is not materialized.
20. A system, comprising:
one or more processors to:
initiate, within a first execution node of an execution platform, a first operator in a query plan to process a set of data and generate an intermediate result of a query;
determine whether the first operator has produced an output from processing the set of data;
responsive to determining that the first operator has generated some output:
push, during execution of the query plan, the output of the first operator to a plurality of secondary operators in the query plan for concurrent processing by the plurality of secondary operators; and
initiate each of the other secondary operators to process the intermediate result to generate a plurality of second results;
operate on the intermediate result to generate a final result; and
store the final result to disk storage within a storage platform that is separate from the execution platform.
21. The system of claim 20 , wherein the first result and the plurality of second results are generated without writing the result data to disk storage.
22. The system of claim 20 , wherein each of the plurality of secondary operators process the intermediate result with a different operation.
23. The system of claim 22 , further comprising delaying operation of at least one of the plurality of secondary operators so as to coordinate timing among other secondary operators of the plurality of secondary operators.
24. The system of claim 20 , wherein the intermediate result generated by the first operator is not recomputed and is processed by the plurality of secondary operators to execute a plurality of different queries.
25. The system of claim 20 , wherein each of the plurality of secondary operators are unique operators.
26. The system of claim 20 , wherein the intermediate result is not materialized.