Parallel audit cycles between primary and secondary event feeds
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising completing a first audit for a primary event type, the first audit generating a set of primary events and completing a second audit for a secondary event type, the second audit generating a draft set of secondary events and an auxiliary feed of un-joined secondary events. The method then performs a join audit check on the auxiliary feed of un-joined secondary events and a set of flags, each flag in the set of flags indicating that a respective un-joined secondary event was properly joined. Based on the results of the join audit check, the method replays a subset of the un-joined secondary events in the auxiliary feed upon determining that the join audit check failed.
1. A method comprising:
loading, by a processor, a plurality of events and a plurality of un-joined events from a data storage device;
completing, by the processor, an audit for the events, the audit comprising processing the events and the plurality of un-joined events using a stream processor to generate an auxiliary feed of un-joined events;
performing, by the processor, a join audit check on the auxiliary feed and a set of flags, a flag in the set of flags indicating that a respective un-joined event was properly joined;
receiving, by the processor, a command to replay a subset of the un-joined events in the auxiliary feed;
loading, by the processor, the subset of the un-joined events in the auxiliary feed; and
replaying, by the processor, the subset of the un-joined events read from the auxiliary feed through the stream processor.
2. The method of claim 1 , wherein completing the audit for the event comprises:
completing a first audit for a primary event type, the first audit generating a set of primary events; and
completing a second audit for a secondary event type, the second audit generating a draft set of secondary events and the auxiliary feed of un-joined events.
3. The method of claim 2 , further comprising publishing the set of primary events after completing the first audit.
4. The method of claim 2 , further comprising certifying the draft set of secondary events upon determining that the join audit check passed.
5. The method of claim 1 , wherein performing a join audit check further comprises performing an inner join operation on the auxiliary feed and the set of flags.
6. The method of claim 5 , wherein performing an inner join operation comprises performing an inner join based on a shared join key.
7. The method of claim 5 , wherein replaying the subset of the un-joined events comprises adding a result set of the inner join operation in a replay queue.
8. The method of claim 1 , wherein performing the join audit check further comprises identifying a dirty interval, a dirty interval comprising an interval in a join window that includes one or more raised flags in the set of flags for a corresponding un-joined event.
9. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
loading a plurality of events and a plurality of un-joined events comprising from a data storage device;
completing an audit for the events, the audit comprising processing the events and the plurality of un-joined events using a stream processor to generate an auxiliary feed of un-joined events;
performing a join audit check on the auxiliary feed and a set of flags, a flag in the set of flags indicating that a respective un-joined event was properly joined; and
receiving, a command to replay a subset of the un-joined events in the auxiliary feed;
loading, the subset of the un-joined events in the auxiliary feed; and
replaying the subset of the un-joined events read from the auxiliary feed through the stream processor.
10. The non-transitory computer-readable storage medium of claim 9 , wherein completing the audit for the event comprises:
completing a first audit for a primary event type, the first audit generating a set of primary events; and
completing a second audit for a secondary event type, the second audit generating a draft set of secondary events and the auxiliary feed of un-joined events.
11. The non-transitory computer-readable storage medium of claim 10 , further comprising publishing the set of primary events after completing the first audit.
12. The non-transitory computer-readable storage medium of claim 10 , further comprising certifying the draft set of secondary events upon determining that the join audit check passed.
13. The non-transitory computer-readable storage medium of claim 9 , wherein performing a join audit check further comprises performing an inner join operation on the auxiliary feed and the set of flags.
14. The non-transitory computer-readable storage medium of claim 13 , wherein performing an inner join operation comprises performing an inner join based on a shared join key.
15. The non-transitory computer-readable storage medium of claim 13 , wherein replaying the subset of the un-joined events comprises adding a result set of the inner join operation in a replay queue.
16. The non-transitory computer-readable storage medium of claim 9 , wherein performing the join audit check further comprises identifying a dirty interval, a dirty interval comprising an interval in a join window that includes one or more raised flags in the set of flags for a corresponding un-joined event.
17. A device comprising:
a processor configured to:
load a plurality of events and a plurality of un-joined events from a data storage device;
complete an audit for the events, the audit comprising processing the events and the plurality of un-joined events using a stream processor to generate an auxiliary feed of un-joined events;
perform a join audit check on the auxiliary feed and a set of flags, a flag in the set of flags indicating that a respective un-joined event was properly joined;
receive a command to replay a subset of the un-joined events in the auxiliary feed;
load the subset of the un-joined events in the auxiliary feed; and
replay the subset of the un-joined events read from the auxiliary feed through the stream processor.
18. The device of claim 17 , wherein completing the audit for the event comprises:
completing a first audit for a primary event type, the first audit generating a set of primary events; and
completing a second audit for a secondary event type, the second audit generating a draft set of secondary events and the auxiliary feed of un-joined events.
19. The device of claim 17 , wherein performing a join audit check further comprises performing an inner join operation on the auxiliary feed and the set of flags.
20. The device of claim 17 , wherein performing the join audit check further comprises identifying a dirty interval, a dirty interval comprising an interval in a join window that includes one or more raised flags in the set of flags for a corresponding un-joined event.