Method and system for event state management in stream processing
Method, system, and programs for event state management in stream processing. In one example, a batch of events is created from a plurality of input events. The batch is associated with a state and is to be processed in one or more stages. The batch of events is stored in a persistent storage. The state associated with the batch is updated based on results of processing the batch in the one or more stages. The state associated with the batch is retrieved.
1. A method implemented on at least one machine each having at least one processor, storage, and a communication platform connected to a network for event state management in stream processing, the method comprising:
creating a batch of events based on a plurality of events inputted from an external source, wherein the batch of events is associated with a state of processing the batch of events and is to be processed in one or more stages;
storing the batch of events in a persistent storage;
processing the batch of events in the one or more stages by at least one of an online event processing system and an offline event process system;
updating the state associated with the batch of events when the processing of the one or more stages is completed, wherein the state associated with the batch of events comprises one or more sub-states corresponding to the one or more stages;
updating each sub-state of the one or more sub-states to “complete” once the batch of events has been processed in the corresponding stage of the one or more stages; and
updating the state associated with the batch of events to “committed” once the batch of events has been processed in all of the one or more stages.
2. The method of claim 1 , further comprising:
if processing of an event in the batch of events fails, updating the state associated with the batch of events to “failed;”
retrieving the batch of events from the persistent storage; and
updating the state associated with the batch of events to “replayed”.
3. The method of claim 1 , further comprising:
receiving a request to read event data for the batch of events in one of the one or more stages;
retrieving the state associated with the batch of events in the one of the one or more stages;
retrieving the corresponding sub-state in the one of the one or more stages; and
retrieving the requested event data for the batch of events from the persistent storage if the state associated with the batch of events is “committed” or if the corresponding sub-state is “complete”.
4. The method of claim 3 , wherein the event data for the batch of events in the one of the one or more stages is updated based on the result of processing the batch of events in the one of the one or more stages only when all of the events in the batch of events have been processed in the one of the one or more stages.
5. The method of claim 1 , wherein each sub-state of the one or more sub-states is updated only when all of the events in the batch of events have been processed in the corresponding stage of the one or more stages.
6. The method of claim 1 , further comprising:
storing the state associated with the batch of events in another persistent storage separate from the persistent storage for storing the batch of events.
7. The method of claim 1 , wherein the batch of events in the one or more stages are processed by the online event processing system and the offline event process system.
8. A system for event state management in stream processing, comprising:
a processor couple to a memory;
an input event managing unit, executed by the processor, configured to create a batch of events based on a plurality of events inputted from an external source, wherein the batch of events is associated with a state of processing the batch of events and is to be processed in one or more stages;
a batch event storage, executed by the processor, configured to store the batch of events;
at least one of an online event processing system and an offline event process system, executed by the processor, configured to process the batch of events in the one or more stages; and
an event batch write managing unit, executed by the processor, configured to update the state associated with the batch of events when the processing of the one or more stages is completed, wherein the state associated with the batch of events comprises one or more sub-states corresponding to the one or more stages, and the event batch write managing unit is further configured to
update each sub-state of the one or more sub-states to “complete” once the batch of events has been processed in the corresponding stage of the one or more stages, and
update the state associated with the batch of events to “committed” once the batch of events has been processed in all of the one or more stages.
9. The system of claim 8 , wherein, if processing of an event in the batch of events fails, the input event managing unit is further configured to:
update the state associated with the batch of events to “failed;”
retrieve the batch of events from the persistent storage; and
update the state associated with the batch of events to “replayed”.
10. The system of claim 8 , further comprising an event batch read managing unit, executed by the processor, configured to:
receive a request to read event data for the batch of events in one of the one or more stages;
retrieve the state associated with the batch of events in the one of the one or more stages;
retrieve the corresponding sub-state in the one of the one or more stages; and
retrieve the requested event data for the batch of events from the persistent storage if the state associated with the batch of events is “committed” or if the corresponding sub-state of the one or more sub-stages is “complete”.
11. The system of claim 10 , wherein the event data for the batch of events in the one of the one or more stages is updated based on a result of processing the batch of events in the one of the one or more stages only when all of the events in the batch of events have been processed in the one of the one or more stages.
12. The system of claim 8 , wherein each sub-state of the one or more sub-states is updated when all of the events in the batch of events have been processed in the corresponding stage of the one or more stages.
13. The system of claim 8 , further comprising a batch state storage, executed by the processor, configured to:
store the state associated with the batch of events separate from the batch event storage for storing the batch of events.
14. A machine-readable tangible and non-transitory medium having information recorded thereon for event state management in stream processing, wherein the information, when read by the machine, causes the machine to perform the following:
creating a batch of events based on a plurality of events inputted from an external source, wherein the batch of events is associated with a state of processing the batch of events and is to be processed in one or more stages;
storing the batch of events in a persistent storage;
processing the batch of events in the one or more stages by at least one of an online event processing system and an offline event process system;
updating the state associated with the batch of events when the processing of the one or more stages is completed, wherein the state associated with the batch of events comprises one or more sub-states corresponding to the one or more stages;
updating each sub-state of the one or more sub-states to “complete” once the batch of events has been processed in the corresponding stage of the one or more stages; and
updating the state associated with the batch of events to “committed” once the batch of events has been processed in all of the one or more stages.
15. The medium of claim 14 , further comprising:
receiving a request to read event data for the batch of events in one of the one or more stages;
retrieving the state associated with the batch of events in the one of the one or more stages; retrieving the corresponding sub-state in the one of the one or more stages; and
retrieving the requested event data for the batch of events from the persistent storage if the state associated with the batch of events is “committed” or if the corresponding sub-state is “complete”.
16. The medium of claim 15 , wherein the event data for the batch of events in the one of the one or more stages is updated based on a result of processing the batch of events in the one of the one or more stages only when all of the events in the batch of events have been processed in the one of the one or more stages.
17. The medium of claim 14 , wherein each sub-state of the one or more sub-states is updated when all of the events in the batch of events have been processed in the corresponding stage.