Parallel audit cycles between primary and secondary event feeds
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising completing a first audit for a primary event type, the first audit generating a set of primary events and completing a second audit for a secondary event type, the second audit generating a draft set of secondary events and an auxiliary feed of un-joined secondary events. The method then performs a join audit check on the auxiliary feed of un-joined secondary events and a set of flags, each flag in the set of flags indicating that a respective un-joined secondary event was properly joined. Based on the results of the join audit check, the method replays a subset of the un-joined secondary events in the auxiliary feed upon determining that the join audit check failed.
1. A method comprising:
completing a first audit for a primary event type, the first audit generating a set of primary events;
completing a second audit for a secondary event type, the second audit generating a draft set of secondary events and an auxiliary feed of un-joined secondary events;
performing a join audit check on the auxiliary feed of un-joined secondary events and a set of flags, each flag in the set of flags indicating that a respective un-joined secondary event was properly joined; and
replaying a subset of the un-joined secondary events in the auxiliary feed upon determining that the join audit check failed.
2. The method of claim 1 , further comprising publishing the set of primary events after completing the first audit.
3. The method of claim 1 , further comprising certifying the draft set of secondary events upon determining that the audit check passed.
4. The method of claim 1 , the performing a join audit check further comprising performing an inner join operation on the auxiliary feed and the set of flags.
5. The method of claim 4 , the performing an inner join operation comprising performing an inner join based on a shared join key.
6. The method of claim 4 , the replaying the subset of the un-joined secondary events comprising adding a result set of the inner join operation in a replay queue.
7. The method of claim 1 , the performing the join audit check further comprising identifying a dirty interval, a dirty interval comprising an interval in a join window that includes one or more raised flags in the set of flags for a corresponding un-joined secondary event.
8. An apparatus comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform the operations of:
completing a first audit for a primary event type, the first audit generating a set of primary events;
completing a second audit for a secondary event type, the second audit generating a draft set of secondary events and an auxiliary feed of un-joined secondary events;
performing a join audit check on the auxiliary feed of un-joined secondary events and a set of flags, each flag in the set of flags indicating that a respective un-joined secondary event was properly joined; and
replaying a subset of the un-joined secondary events in the auxiliary feed upon determining that the join audit check failed.
9. The apparatus of claim 8 , the operations further comprising publishing the set of primary events after completing the first audit.
10. The apparatus of claim 8 , the operations further comprising certifying the draft set of secondary events upon determining that the audit check passed.
11. The apparatus of claim 8 , the performing a join audit check further comprising performing an inner join operation on the auxiliary feed and the set of flags.
12. The apparatus of claim 11 , the performing an inner join operation comprising performing an inner join based on a shared join key.
13. The apparatus of claim 11 , the replaying the subset of the un-joined secondary events comprising adding a result set of the inner join operation in a replay queue.
14. The apparatus of claim 8 , the performing the join audit check further comprising identifying a dirty interval, a dirty interval comprising an interval in a join window that includes one or more raised flags in the set of flags for a corresponding un-joined secondary event.
15. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining the steps of:
completing a first audit for a primary event type, the first audit generating a set of primary events;
completing a second audit for a secondary event type, the second audit generating a draft set of secondary events and an auxiliary feed of un joined secondary events;
performing a join audit check on the auxiliary feed of un joined secondary events and a set of flags, each flag in the set of flags indicating that a respective un-joined secondary event was properly joined; and
replaying a subset of the un-joined secondary events in the auxiliary feed upon determining that the join audit check failed.
16. The computer-readable storage medium of claim 15 , the instructions further defining the step of publishing the set of primary events after completing the first audit.
17. The computer-readable storage medium of claim 15 , the instructions further defining the step of certifying the draft set of secondary events upon determining that the audit check passed.
18. The computer-readable storage medium of claim 15 , the performing a join audit check further comprising performing an inner join operation on the auxiliary feed and the set of flags.
19. The computer-readable storage medium of claim 18 , the performing an inner join operation comprising performing an inner join based on a shared join key.
20. The computer-readable storage medium of claim 18 , the replaying the subset of the un-joined secondary events comprising adding a result set of the inner join operation in a replay queue.