Providing strong ordering in multi-stage streaming processing
The technology disclosed relates to providing strong ordering in multi-stage processing of near real-time (NRT) data streams. In particular, it relates to maintaining current batch-stage information for a batch at a grid-scheduler in communication with a grid-coordinator that controls dispatch of batch-units to the physical threads for a batch-stage. This includes operating a computing grid, and queuing data from the NRT data streams as batches in pipelines for processing over multiple stages in the computing grid. Also included is determining, for a current batch-stage, batch-units pending dispatch, in response to receiving the current batch-stage information; identifying physical threads that processed batch-units for a previous batch-stage on which the current batch-stage depends and have registered pending tasks for the current batch-stage; and dispatching the batch-units for the current batch-stage to the identified physical threads subsequent to complete processing of the batch-units for the previous batch-stage.
1. A method, comprising:
maintaining, by a grid scheduler, current batch stage information comprising current batch units and downstream batch units that depend on completion of the current batch units;
determining, by the grid scheduler, a batch unit pending dispatch from the downstream batch units for a current batch stage identified in the current batch stage information;
identifying, by the grid scheduler, one or more physical threads in a computing grid that processed batch units for the current batch stage on which the batch unit pending dispatch depends and have registered pending tasks for the current batch stage; and
causing, by the grid scheduler, a dispatch of the batch unit pending dispatch to the one or more identified physical threads subsequent to complete processing of the batch units for the current batch stage,
wherein at least one of the maintaining, determining, identifying, and causing are performed by one or more computers.
2. The method of claim 1 , the determining further comprising:
assigning, by the grid scheduler, priority levels to the current batch units; and
determining, by the grid scheduler, the batch unit pending dispatch based on the priority levels.
3. The method of claim 1 , further comprising:
queuing, by the grid scheduler, a batch in a pipeline in a plurality of pipelines, wherein the pipeline processes the batch over multiple stages in the computing grid.
4. The method of claim 1 , further comprising:
assigning, by the grid scheduler, unique batch identifiers to the current batch units; and
communicating, by the grid scheduler, the unique batch identifiers to a grid controller.
5. The method of claim 1 , further comprising:
assigning, by the grid scheduler, a stage identifier to each processing stage of the batch unit pending dispatch; and
communicating, by the grid scheduler, the stage identifier to a grid controller.
6. The method of claim 1 , further comprising:
assigning, by the grid scheduler, a pipeline identifier to a pipeline of a plurality of pipelines in the computing grid; and
communicating, by the grid scheduler, the pipeline identifier to a grid controller.
7. The method of claim 1 , further comprising:
determining, by the grid scheduler, a second batch unit that is dependent on the batch unit pending dispatch; and
delaying, by the grid scheduler, dispatch of the second batch unit until the batch unit pending dispatch completes.
8. A system, comprising:
a memory; and
at least one processor coupled to the memory and configured to:
maintain current batch stage information comprising current batch units and downstream batch units that depend on completion of the current batch units at a grid scheduler;
determine a batch unit pending dispatch from the downstream batch units for a current batch stage identified in the current batch stage information;
identify one or more physical threads in a computing grid that processed batch units for the current batch stage on which the batch unit pending dispatch depends and have registered pending tasks for the current batch stage; and
cause a dispatch of the batch unit pending dispatch to the one or more identified physical threads subsequent to complete processing of the batch units for the current batch stage.
9. The system of claim 8 , the at least one processor further configured to:
assign priority levels to the current batch units; and
determine the batch unit pending dispatch based on the priority levels.
10. The system of claim 8 , the at least one processor further configured to:
queue a batch in a pipeline in a plurality of pipelines, wherein the pipeline processes the batch over multiple stages in the computing grid.
11. The system of claim 8 , the at least one processor further configured to:
assign unique batch identifiers to the current batch units; and
communicate the unique batch identifiers to a grid controller.
12. The system of claim 8 , the at least one processor further configured to:
assign a stage identifier to each processing stage of the batch unit pending dispatch; and
communicate the stage identifier to a grid controller.
13. The system of claim 8 , the at least one processor further configured to:
assign a pipeline identifier to a pipeline of a plurality of pipelines in the computing grid; and
communicate the pipeline identifier to a grid controller.
14. The system of claim 8 , the at least one processor further configured to:
determine a second batch unit that is dependent on the batch unit pending dispatch; and
delay dispatch of the second batch unit until the batch unit pending dispatch completes.
15. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:
maintaining current batch stage information comprising current batch units and downstream batch units that depend on completion of the current batch units at a grid scheduler;
determining a batch unit pending dispatch from the downstream batch units for a current batch stage identified in the current batch stage information;
identifying one or more physical threads in a computing grid that processed batch units for the current batch stage on which the batch unit pending dispatch depends and have registered pending tasks for the current batch stage; and
causing a dispatch of the batch unit pending dispatch to the one or more identified physical threads subsequent to complete processing of the batch units for the current batch stage.
16. The non-transitory computer-readable device of claim 15 , the operations further comprising:
assigning priority levels to the current batch units; and
determining the batch unit pending dispatch based on the priority levels.
17. The non-transitory computer-readable device of claim 15 , the operations further comprising:
queuing a batch in a pipeline in a plurality of pipelines, wherein the pipeline processes the batch over multiple stages in the computing grid.
18. The non-transitory computer-readable device of claim 15 , the operations further comprising:
assigning unique batch identifiers to the current batch units; and
communicating the unique batch identifiers to a grid controller.
19. The non-transitory computer-readable device of claim 15 , the operations further comprising:
assigning a stage identifier to each processing stage of the batch unit pending dispatch; and
communicating the stage identifier to a grid controller.
20. The non-transitory computer-readable device of claim 15 , the operations further comprising:
assigning a pipeline identifier to a pipeline of a plurality of pipelines in the computing grid; and
communicating the pipeline identifier to a grid controller.