Exactly-once performance from a streaming pipeline in a fault-vulnerable system
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving a stream of events, where each event comprises a header including an event type field and a timestamp. and identifying that the header of a given event in the stream of events includes at least one secondary event in a list of secondary identifiers included in the header. A flag is set in a distributed database for the at least one secondary event, then writing the given event to a table in the distributed database occurs, the table selected based on a respective event type field and timestamp associated with the given event. Deduplicating the given event and the at least one secondary event in response to a close of books (COB) signal occurs, after which writing at least one deduplicated event to a storage device occurs, the at least one deduplicated event selected from the group consisting of the given event and the at least one secondary event.
1. A method comprising:
receiving a stream of events, each event comprising a header including an event type field and a timestamp;
identifying that the header of a given event in the stream of events includes at least one secondary event in a list of secondary identifiers included in the header;
setting a flag in a distributed database for the at least one secondary event;
writing the given event to a table in the distributed database, the table selected based on a respective event type field and timestamp associated with the given event;
deduplicating the given event and the at least one secondary event in response to a close of books (COB) signal; and
writing at least one deduplicated event to a storage device, the at least one deduplicated event selected from the group consisting of the given event and the at least one secondary event.
2. The method of claim 1 , further comprising executing a wtable creation process to periodically create a set of new tables in the distributed database.
3. The method of claim 1 , further comprising executing a table pruning process to periodically discard unused tables in the distributed database.
4. The method of claim 1 , the receiving a stream of events comprising receiving a plurality of messages from a streaming queue.
5. The method of claim 4 , further comprising extracting the events from the plurality of messages.
6. The method of claim 1 , the setting the flag for the at least one secondary event comprising setting the flag for the at least one secondary event stored in a second table.
7. The method of claim 1 , further comprising deferring the writing the given event until the setting the flag is completed.
8. The method of claim 1 , further comprising detecting that the given event is a duplicate event of a previously written event written to the table of the distributed database.
9. The method of claim 8 , further comprising writing the given event to a cell associated with the previously written event and updating a version identifier of the given event.
10. The method of claim 1 , the deduplicating the given event and the at least one secondary event comprising:
determining that the at least one secondary event includes multiple versions; and
selecting a most recent version of the at least one secondary event as the at least one deduplicated event.
11. The method of claim 10 , the deduplicating the given event and the at least one secondary event comprising replaying the at least one secondary event if the at least one secondary event includes the flag.
12. The method of claim 11 , further comprising setting a join status of the most recent version of the at least one of the plurality of events as failed if the at least one secondary event does not include the flag and was not replayed.
13. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:
receiving a stream of events, each event comprising a header including an event type field and a timestamp;
identifying that the header of a given event in the stream of events includes at least one secondary event in a list of secondary identifiers included in the header;
setting a flag in a distributed database for the at least one secondary event;
writing the given event to a table in the distributed database, the table selected based on a respective event type field and timestamp associated with the given event;
deduplicating the given event and the at least one secondary event in response to a close of books (COB) signal; and
writing at least one deduplicated event to a storage device, the at least one deduplicated event selected from the group consisting of the given event and the at least one secondary event.
14. The computer-readable storage medium of claim 13 , the setting the flag for the at least one secondary event comprising setting the flag for the at least one secondary event stored in a second table.
15. The computer-readable storage medium of claim 13 , the computer program instructions further defining the step of deferring the writing the given event until the setting the flag is completed.
16. The computer-readable storage medium of claim 13 , further comprising detecting that the given event is a duplicate event of a previously written event written to the table of the distributed database.
17. The computer-readable storage medium of claim 13 , the deduplicating the given event and the at least one secondary event comprising:
determining that the at least one secondary event includes multiple versions; and
selecting a most recent version of the at least one secondary event as the at least one deduplicated event.
18. The computer-readable storage medium of claim 17 , the deduplicating the given event and the at least one secondary event comprising replaying the at least one secondary event if the at least one secondary event includes the flag.
19. The computer-readable storage medium of claim 18 , the computer program instructions further defining the step of setting a join status of the most recent version of the at least one of the plurality of events as failed if the at least one secondary event does not include the flag and was not replayed.
20. A device comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:
receiving a stream of events, each event comprising a header including an event type field and a timestamp,
identifying that the header of a given event in the stream of events includes at least one secondary event in a list of secondary identifiers included in the header,
setting a flag in a distributed database for the at least one secondary event,
writing the given event to a table in the distributed database, the table selected based on a respective event type field and timestamp associated with the given event,
deduplicating the given event and the at least one secondary event in response to a close of books (COB) signal, and
writing at least one deduplicated event to a storage device, the at least one deduplicated event selected from the group consisting of the given event and the at least one secondary event.