Exactly-once performance from a streaming pipeline in a fault-vulnerable system
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving a stream of events, where each event comprises a header including an event type field and a timestamp. and identifying that the header of a given event in the stream of events includes at least one secondary event in a list of secondary identifiers included in the header. A flag is set in a distributed database for the at least one secondary event, then writing the given event to a table in the distributed database occurs, the table selected based on a respective event type field and timestamp associated with the given event. Deduplicating the given event and the at least one secondary event in response to a close of books (COB) signal occurs, after which writing at least one deduplicated event to a storage device occurs, the at least one deduplicated event selected from the group consisting of the given event and the at least one secondary event.
1. A method comprising:
receiving an event, the event having a header that includes at least one secondary event in a list of secondary identifiers included in the header;
writing the event to a table in a database, the table selected based on a respective event type field and timestamp associated with the event;
deduplicating the event and the at least one secondary event in response to a close of books (COB) signal;
selecting one of the event or the at least one secondary event for writing, as a deduplicated event, to a storage device; and
writing the deduplicated event to the storage device.
2. The method of claim 1 , further comprising executing a table creation process to periodically create a set of new tables in the database.
3. The method of claim 1 , further comprising executing a table pruning process to periodically discard unused tables in the database.
4. The method of claim 1 , the receiving a stream of events comprising receiving a plurality of messages from a streaming queue.
5. The method of claim 4 , further comprising extracting the events from the plurality of messages.
6. The method of claim 1 , further comprising setting a flag for the at least one secondary event comprising setting the flag for the at least one secondary event stored in a second table.
7. The method of claim 6 , further comprising deferring the writing the event until the setting the flag is completed.
8. The method of claim 1 , further comprising detecting that the event is a duplicate event of a previously written event written to the table of the database.
9. The method of claim 8 , further comprising writing the event to a cell associated with the previously written event and updating a version identifier of the event.
10. The method of claim 1 , the deduplicating the event and the at least one secondary event comprising:
determining that the at least one secondary event includes multiple versions; and
selecting a most recent version of the at least one secondary event as the at least one deduplicated event.
11. The method of claim 10 , the deduplicating the event and the at least one secondary event comprising replaying the at least one secondary event if the at least one secondary event includes a flag.
12. The method of claim 11 , further comprising setting a join status of the most recent version of the event as failed if the at least one secondary event does not include the flag and was not replayed.
13. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
receiving an event, the event having a header that includes at least one secondary event in a list of secondary identifiers included in the header;
writing the event to a table in a database, the table selected based on a respective event type field and timestamp associated with the event;
deduplicating the event and the at least one secondary event in response to a close of books (COB) signal;
selecting one of the event or the at least one secondary event for writing as a deduplicated event to a storage device; and
writing the deduplicated event to the storage device.
14. The non-transitory computer-readable storage medium of claim 13 , further comprising executing a table creation process to periodically create a set of new tables in the database.
15. The non-transitory computer-readable storage medium of claim 13 , further comprising executing a table pruning process to periodically discard unused tables in the database.
16. The non-transitory computer-readable storage medium of claim 13 , the receiving a stream of events comprising receiving a plurality of messages from a streaming queue.
17. The non-transitory computer-readable storage medium of claim 13 , further comprising setting a flag for the at least one secondary event comprising setting the flag for the at least one secondary event stored in a second table.
18. The non-transitory computer-readable storage medium of claim 13 , further comprising detecting that the event is a duplicate event of a previously written event written to the table of the database.
19. The non-transitory computer-readable storage medium of claim 13 , the deduplicating the event and the at least one secondary event comprising:
determining that the at least one secondary event includes multiple versions; and
selecting a most recent version of the at least one secondary event as the at least one deduplicated event.
20. A device comprising:
a processor; and
a storage medium for tangibly storing thereon logic for execution by the processor, the logic comprising instructions for:
receiving an event, the event having a header that includes at least one secondary event in a list of secondary identifiers included in the header;
writing the event to a table in a database, the table selected based on a respective event type field and timestamp associated with the event;
deduplicating the event and the at least one secondary event in response to a close of books (COB) signal;
selecting one of the event or the at least one secondary event for writing as a deduplicated event to a storage device; and
writing the deduplicated event to the storage device.