Opportunistic cloud data platform pipeline scheduler
Methods, systems, and computer programs are presented for scheduling and executing request plans using an opportunistic approach. An opportunistic scheduler generates a request plan for a request on a cloud data platform, the request plan comprising a plurality of operations and identifies a plurality of contingent operations from the plurality of operations of the request plan. The opportunistic scheduler schedules the plurality of contingent operations for execution and sets the scheduled plurality of contingent operations to execute at a specific position in the request plan. The opportunistic scheduler sets remaining operations for execution by any available thread as threads that are processing the request plan become available and processes the request plan according to the scheduled plurality of contingent operations.
1. A method comprising:
generating a request plan for a request on a cloud data platform, the request plan comprising a plurality of operations;
identifying a plurality of contingent operations from the plurality of operations of the request plan;
scheduling, by at least one hardware processor, the plurality of contingent operations for execution using a scheduler;
setting the scheduled plurality of contingent operations to execute at a specific position in the request plan;
setting remaining operations for execution by any available thread as threads that are processing the request plan become available; and
processing the request plan according to the scheduled plurality of contingent operations.
2. The method of claim 1 , wherein the cloud data platform includes a plurality of execution nodes on which the plurality of operations is performed, each execution node of the plurality of execution nodes includes the scheduler configured to select an operator of a plurality of operators to receive the request plan.
3. The method of claim 2 , wherein the scheduler further comprises:
synchronizing distributed execution of the plurality of operators across all worker processes; and
finalizing the plurality of operators in a deterministic manner for correct output data.
4. The method of claim 1 , further comprising:
listing, by the scheduler, one or more pipelines as ready for processing, a pipeline of the one or more pipelines comprising an additional plurality of operations of the request plan.
5. The method of claim 4 , further comprising:
executing the one or more pipelines while an existing pipeline that comprises the plurality of operations is executing on an execution node of the cloud data platform.
6. The method of claim 4 , wherein each pipeline of the one or more pipelines is a sequence of connected row set operators (RSOs), wherein each sequence of connected RSOs start and end with one or more leaf operators, end in a pipeline breaker, or end in a result operator.
7. The method of claim 1 , further comprising:
restarting one or more of the plurality of operations without restarting other operations in the request plan.
8. A system comprising:
one or more hardware processors of a machine; and
at least one memory storing instructions that, when executed by the one or more hardware processors, cause the system to perform operations comprising:
generating a request plan for a request on a cloud data platform, the request plan comprising a plurality of operations;
identifying a plurality of contingent operations from the plurality of operations of the request plan;
scheduling, by at least one hardware processor, the plurality of contingent operations for execution using a scheduler;
setting the scheduled plurality of contingent operations to execute at a specific position in the request plan;
setting remaining operations for execution by any available thread as threads that are processing the request plan become available; and
processing the request plan according to the scheduled plurality of contingent operations.
9. The system of claim 8 , wherein the cloud data platform includes a plurality of execution nodes on which the plurality of operations is performed, each execution node of the plurality of execution nodes includes the scheduler configured to select an operator of a plurality of operators to receive the request plan.
10. The system of claim 9 , wherein the scheduler further comprises:
synchronizing distributed execution of the plurality of operators across all worker processes; and
finalizing the plurality of operators in a deterministic manner for correct output data.
11. The system of claim 8 , further comprising:
listing, by the scheduler, one or more pipelines as ready for processing, a pipeline of the one or more pipelines comprising an additional plurality of operations of the request plan.
12. The system of claim 11 , further comprising:
executing the one or more pipelines while an existing pipeline that comprises the plurality of operations is executing on an execution node of the cloud data platform.
13. The system of claim 11 , wherein each pipeline of the one or more pipelines is a sequence of connected row set operators (RSOs), wherein each sequence of connected RSOs start and end with one or more leaf operators, end in a pipeline breaker, or end in a result operator.
14. The system of claim 8 , further comprising:
restarting one or more of the plurality of operations without restarting other operations in the request plan.
15. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
generating a request plan for a request on a cloud data platform, the request plan comprising a plurality of operations;
identifying a plurality of contingent operations from the plurality of operations of the request plan;
scheduling, by at least one hardware processor, the plurality of contingent operations for execution using a scheduler;
setting the scheduled plurality of contingent operations to execute at a specific position in the request plan;
setting remaining operations for execution by any available thread as threads that are processing the request plan become available; and
processing the request plan according to the scheduled plurality of contingent operations.
16. The machine-storage medium of claim 15 , wherein the cloud data platform includes a plurality of execution nodes on which the plurality of operations is performed, each execution node of the plurality of execution nodes includes the scheduler configured to select an operator of a plurality of operators to receive the request plan.
17. The machine-storage medium of claim 16 , wherein the scheduler further comprises:
synchronizing distributed execution of the plurality of operators across all worker processes; and
finalizing the plurality of operators in a deterministic manner for correct output data.
18. The machine-storage medium of claim 15 , further comprising:
listing, by the scheduler, one or more pipelines as ready for processing, a pipeline of the one or more pipelines comprising an additional plurality of operations of the request plan; and
executing the one or more pipelines while an existing pipeline that comprises the plurality of operations is executing on an execution node of the cloud data platform.
19. The machine-storage medium of claim 18 , wherein each pipeline of the one or more pipelines is a sequence of connected row set operators (RSOs), wherein each sequence of connected RSOs start and end with one or more leaf operators, end in a pipeline breaker, or end in a result operator.
20. The machine-storage medium of claim 19 , further comprising:
restarting one or more of the plurality of operations without restarting other operations in the request plan.