Push model for intermediate query results
A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.
1. A method comprising:
initiating, within a first processor, a first parallel execution process in a query plan to process a set of data and generate an intermediate result of a query;
providing, by the first processor, during execution of the query plan, the intermediate result to a second set of processors for processing by a plurality of secondary parallel execution processes within the second set of processors, wherein each of the first set of processors and the second set of processors is decoupled from a data storage device that stores the set of data prior to the execution of the query plan;
initiating, by the second set of processors, the plurality of secondary parallel execution processes to process the intermediate result to generate a plurality of secondary results; and
storing a final result to storage within a storage platform.
2. The method of claim 1 , wherein the plurality of secondary results generated without writing the intermediate result to the data storage device.
3. The method of claim 1 , wherein each of the plurality of secondary parallel execution processes processes the intermediate result using a different operation.
4. The method of claim 1 , further comprising:
delaying operation of at least one of the plurality of secondary parallel execution processes so as to coordinate timing among other secondary parallel execution processes of the plurality of secondary parallel execution processes.
5. The method of claim 1 , wherein the intermediate result comprises a plurality of rows of database data.
6. The method of claim 1 , wherein each of the plurality of secondary parallel execution processes are unique operators.
7. The method of claim 1 , wherein the intermediate result is not materialized.
8. The method of claim 1 , wherein the intermediate result generated by the first parallel execution processes is not materialized to a temporary structure.
9. The method of claim 1 , wherein the each of the first processor and the second set of one or more processors is coupled to the data storage device via a communications network.
10. The method of claim 1 , wherein the each of the first set of one or more processors and the second set of one or more processors includes a local cache.
11. A system comprising:
a first set of processors programmed to:
initiate a first parallel execution process in a query plan to process a set of data and generate an intermediate result of a query, and
provide, during execution of the query plan, the intermediate result to a second set of processors for processing by the second set of processors, wherein each of the first set of processors and the second set of processors is decoupled from a data storage device that stores the set of data prior to the execution of the query plan; and
the second set of processors programmed to initiate a plurality of secondary parallel execution processes to process the intermediate result to generate a plurality of secondary results; and
a storage platform programmed to store a final result to storage.
12. The system of claim 11 , wherein the plurality of secondary results generated without writing the intermediate result to the data storage device.
13. The system of claim 11 , wherein each of the plurality of secondary parallel execution processes processes the intermediate result using a different operation.
14. The system of claim 11 , the first set of processors further programmed to:
delay operation of at least one of the plurality of secondary parallel execution processes so as to coordinate timing among other secondary parallel execution processes of the plurality of secondary parallel execution processes.
15. The system of claim 11 , wherein the intermediate result comprises a plurality of rows of database data.
16. The system of claim 11 , wherein each of the plurality of secondary parallel execution processes are unique operators.
17. The system of claim 11 , wherein the intermediate result is not materialized.
18. The system of claim 11 , wherein the intermediate result generated by the first parallel execution processes is not materialized to a temporary structure.
19. The system of claim 11 , wherein the each of the first processor and the second set of one or more processors are coupled to the data storage device via a communications network.
20. The system of claim 11 , wherein the each of the first set of processors and the second set of one or more processors includes a local cache.
21. A non-transitory computer-readable medium storing instructions which, when executed by one or more third set of one or more processors of a computing system, cause the computing system to:
initiate, within a first processor, a first parallel execution process in query plan to process set of data and generate an intermediate result of query;
provide, by the first processor, during execution of the query plan, the intermediate result to a second set of processors for processing by a plurality of secondary parallel execution processes by the second set of processors, wherein each of the first set and the second set of processors are decoupled from a data storage device that stores the set of data prior to the execution of the query plan;
initiate, by the second set of processors, the plurality of secondary parallel execution processes to process the intermediate result to generate a plurality of secondary results; and
store a final result to storage within a storage platform.
22. The non-transitory computer-readable medium of claim 21 , wherein the plurality of secondary results is generated without writing the intermediate result to the data storage device.
23. The non-transitory computer-readable medium of claim 21 , wherein the query coordinating process further to:
delay operation of at least one of the plurality of secondary parallel execution processes so as to coordinate timing among other secondary parallel execution processes of the plurality of secondary parallel execution processes.
24. The non-transitory computer-readable medium of claim 21 , wherein the each of the first processor and the second set of one or more processors is coupled to the data storage device via a communications network.