Push model for scheduling query plans
A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.
1. A method comprising:
processing, by one or more processors, a set of data with a first operator of a query plan to generate an intermediate result of a query; and
in response to generating the intermediate result, transmitting, during execution of the query plan, the intermediate result to a plurality of secondary operators of the query plan, wherein each of the plurality of secondary operators concurrently process the intermediate result with a different type of operation to generate a plurality of second results, and wherein processing of the intermediate result by one or more of the plurality of secondary operators is timed to coordinate the generation of the plurality of second results based on a final operation to be executed on the plurality of second results.
2. The method of claim 1 , further comprising:
executing the final operation on the plurality of second results to generate a final result; and
storing the final result to a storage platform.
3. The method of claim 2 , wherein executing the final operation on the plurality of second results comprises:
performing a join operation on the plurality of second results to generate the final result.
4. The method of claim 2 , wherein the intermediate result and the plurality of second results are not written to the storage platform.
5. The method of claim 1 , wherein the intermediate result generated by the first operator is not recomputed and is processed by each of the plurality of secondary operators to execute a plurality of different queries.
6. The method of claim 2 , wherein the first operator and the plurality of second operators execute on an execution platform comprising a plurality of execution nodes, and wherein the storage platform scales independently of the execution platform.
7. The method of claim 1 , wherein the intermediate result is not materialized.
8. A system comprising:
a memory; and
one or more processors, operatively coupled to the memory, the one or more processors to:
process a set of data with a first operator of a query plan to generate an intermediate result of a query; and
in response to generating the intermediate result, transmit, during execution of the query plan, the intermediate result to a plurality of secondary operators of the query plan, wherein each of the plurality of secondary operators concurrently process the intermediate result with a different type of operation to generate a plurality of second results, and wherein processing of the intermediate result by one or more of the plurality of secondary operators is timed to coordinate the generation of the plurality of second results based on a final operation to be executed on the plurality of second results.
9. The system of claim 8 , wherein the one or more processors are to:
execute the final operation on the plurality of second results to generate a final result; and
store the final result to a storage platform.
10. The system of claim 9 , wherein to execute the final operation on the plurality of second results, the one or more processors are further to:
perform a join operation on the plurality of second results to generate the final result.
11. The system of claim 9 , wherein the one or more processors do not write the intermediate result and the plurality of second results to the storage platform.
12. The system of claim 8 , wherein the one or more processors do not recompute the intermediate result generated by the first operator, and process, by each of the plurality of secondary operators, the intermediate result to execute a plurality of different queries.
13. The system of claim 9 , wherein the one or more processors execute the first operator and the plurality of second operators on an execution platform comprising a plurality of execution nodes, and wherein the storage platform scales independently of the execution platform.
14. The system of claim 8 , wherein the one or more processors do not materialize the intermediate result.
15. A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors, cause the one or more processors to:
process a set of data with a first operator of a query plan to generate an intermediate result of a query; and
in response to generating the intermediate result, transmit, during execution of the query plan, the intermediate result to a plurality of secondary operators of the query plan, wherein each of the plurality of secondary operators concurrently process the intermediate result with a different type of operation to generate a plurality of second results, and wherein processing of the intermediate result by one or more of the plurality of secondary operators is timed to coordinate the generation of the plurality of second results based on a final operation to be executed on the plurality of second results.
16. The non-transitory computer-readable medium of claim 15 , wherein the one or more processors are to:
execute the final operation on the plurality of second results to generate a final result; and
store the final result to a storage platform.
17. The non-transitory computer-readable medium of claim 16 , wherein to execute the final operation on the plurality of second results, the one or more processors are further to:
perform a join operation on the plurality of second results to generate the final result.
18. The non-transitory computer-readable medium of claim 16 , wherein the one or more processors do not write the intermediate result and the plurality of second results to the storage platform.
19. The non-transitory computer-readable medium of claim 15 , wherein the one or more processors do not recompute the intermediate result generated by the first operator, and process, by each of the plurality of secondary operators, the intermediate result to execute a plurality of different queries.
20. The non-transitory computer-readable medium of claim 16 , wherein the one or more processors execute the first operator and the plurality of second operators on an execution platform comprising a plurality of execution nodes, and wherein the storage platform scales independently of the execution platform.
21. The non-transitory computer-readable medium of claim 15 , wherein the one or more processors do not materialize the intermediate result.