Duplicate detection and replay to ensure exactly-once delivery in a streaming pipeline
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising processing a plurality of events using a stream processor and executing a deduplication process on the plurality of events using the stream processor. The plurality of events is outputted to a streaming queue and a close of books (COB) of a data transport is detected. Then, an audit process is initiated in response to detecting the COB signal, the audit process comprising comparing a set of raw events to a set of events in the streaming queue to identify a set of missing events, and replaying a set of missing events through the stream processor.
1. A method comprising:
processing a plurality of events using a stream processor;
executing a deduplication process on the plurality of events using the stream processor;
outputting the plurality of events to a streaming queue; and
initiating an audit process in response to detecting a close of books (COB) signal comprising a predetermined time interval, the audit process comprising:
comparing a set of raw events to a set of events in the streaming queue to identify a set of missing events, and
replaying a set of missing events through the stream processor, the set of missing events configured to bypass the deduplication process while being re-processed.
2. The method of claim 1 , the executing the deduplication process comprising:
querying a data store using an event in the plurality of events;
dropping the event if the event exists in the data store; and
writing the event to the data store if the event does not exist in the data store.
3. The method of claim 2 , the querying the data store comprising querying a distributed database.
4. The method of claim 1 , the replaying the set of missing events comprising annotating the missing events with deduplication keys.
5. The method of claim 4 , the executing the deduplication process comprising dropping an event in the set of missing events based on a corresponding deduplication key.
6. The method of claim 1 , the replaying the set of missing events comprising:
incrementing an audit replay level; and
annotating the set of missing events using the audit replay level.
7. The method of claim 6 , the replaying the set of missing events further comprising adjusting a filter of deduplication process based on the audit replay level.
8. The method of claim 7 , the executing the deduplication process comprising dropping an event using the filter if the event is annotated with a replay level below the audit replay level.
9. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:
processing a plurality of events using a stream processor;
executing a deduplication process on the plurality of events using the stream processor;
outputting the plurality of events to a streaming queue; and
initiating an audit process in response to detecting a close of books (COB) signal comprising a predetermined time interval, the audit process comprising:
comparing a set of raw events to a set of events in the streaming queue to identify a set of missing events, and
replaying a set of missing events through the stream processor, the set of missing events configured to bypass the deduplication process while being re-processed.
10. The computer-readable storage medium of claim 9 , the executing the deduplication process comprising:
querying a data store using an event in the plurality of events;
dropping the event if the event exists in the data store; and
writing the event to the data store if the event does not exist in the data store.
11. The computer-readable storage medium of claim 10 , the querying the data store comprising querying a distributed database.
12. The computer-readable storage medium of claim 9 , the replaying the set of missing events comprising annotating the missing events with deduplication keys.
13. The computer-readable storage medium of claim 12 , the executing the deduplication process comprising dropping an event in the set of missing events based on a corresponding deduplication key.
14. The computer-readable storage medium of claim 9 , the replaying the set of missing events comprising:
incrementing an audit replay level; and
annotating the set of missing events using the audit replay level.
15. The computer-readable storage medium of claim 14 , the replaying the set of missing events further comprising adjusting a filter of deduplication process based on the audit replay level.
16. The computer-readable storage medium of claim 15 , the executing the deduplication process comprising dropping an event using the filter if the event is annotated with a replay level below the audit replay level.
17. A system comprising:
a stream processor for processing a plurality of events, the stream processor further comprising a dedupe bolt for executing a deduplication process on the plurality of events;
a streaming queue to receive the plurality of events processed by the stream processor; and
an auditor configured to initiate an audit process in response to detecting a close of books (COB) signal comprising a predetermined time interval, the audit process comprising:
comparing a set of raw events to a set of events in the streaming queue to identify a set of missing events, and
replaying a set of missing events through the stream processor, the set of missing events configured to bypass the deduplication process while being re-processed.
18. The system of claim 17 , the executing the deduplication process comprising:
querying a data store using an event in the plurality of events;
dropping the event if the event exists in the data store; and
writing the event to the data store if the event does not exist in the data store.
19. The system of claim 17 , the replaying the set of missing events comprising annotating the missing events with deduplication keys.
20. The system of claim 17 , the replaying the set of missing events comprising:
incrementing an audit replay level; and
annotating the set of missing events using the audit replay level.