Certifying events in a streaming pipeline
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising detecting a real close of books (COB) of a data transport, the real COB associated with a set of raw events transmitted over the data transport, flushing a stream processor in response to detecting the real COB, and retrieving a set of processed events from a distributed file system after the flushing is complete. A fact COB computation is then performed on the set of processed events and the set of raw events, the fact COB computation outputting a number of missing events, each missing event representing a raw event that is not present in the set of processed events. The processed events are then certified upon determining that the number of missing events is below a threshold.
1. A method comprising:
detecting a real close of books (COB) of a data transport, the real COB associated with a set of raw events transmitted over the data transport and detected after a preconfigured number of validations have occurred for a corresponding batch time, the preconfigured number of validations corresponding to a ratio between the set of raw events transmitted over the data transport and a second set of events written to a raw event storage device;
flushing a stream processor in response to detecting the real COB;
retrieving a set of processed events from a distributed file system after the flushing is complete;
performing a fact COB computation on the set of processed events and the set of raw events, the set of processed events and the set of raw events cogrouped using shared join keys, the fact COB computation outputting a number of missing events, each missing event representing a raw event that is not cogrouped with a corresponding processed event; and
certifying the processed events upon determining that the number of missing events is below a threshold.
2. The method of claim 1 , further comprising initiating a replay of the missing events if the number of missing events is above a threshold.
3. The method of claim 1 , the detecting the real close of books comprising determining that a ratio of the number of events received over the data transport and the number of events written to a raw event storage is above a preconfigured threshold.
4. The method of claim 1 , further comprising identifying a set of missing events based on the cogrouping, the missing events comprising empty cells in a relation generated by the cogrouping.
5. The method of claim 1 , the flushing the stream processor comprising updating a scoreboard indicating with a replay level associated with a batch interval associated with the real COB.
6. The method of claim 5 , the flushing the stream processor further comprising polling the scoreboard to determine that the flushing is complete.
7. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
detecting a real close of books (COB) of a data transport, the real COB associated with a set of raw events transmitted over the data transport and detected after a preconfigured number of validations have occurred for a corresponding batch time, the preconfigured number of validations corresponding to a ratio between the set of raw events transmitted over the data transport and a second set of events written to a raw event storage device;
flushing a stream processor in response to detecting the real COB;
retrieving a set of processed events from a distributed file system after the flushing is complete;
performing a fact COB computation on the set of processed events and the set of raw events, the set of processed events and the set of raw events comprising cogrouping the processed events and the raw events based on a join key, the fact COB computation outputting a number of missing events, each missing event representing a raw event that is not cogrouped with a corresponding processed events; and
certifying the processed events upon determining that the number of missing events is below a threshold.
8. The computer-readable storage medium of claim 7 , further comprising initiating a replay of the missing events if the number of missing events is above a threshold.
9. The computer-readable storage medium of claim 7 , the detecting the real close of books comprising determining that a ratio of the number of events received over the data transport and the number of events written to a raw event storage is above a preconfigured threshold.
10. The computer-readable storage medium of claim 7 , further comprising identifying a set of missing events based on the cogrouping, the missing events comprising empty cells in a relation generated by the cogrouping.
11. The computer-readable storage medium of claim 7 , the flushing the stream processor comprising updating a scoreboard indicating with a replay level associated with a batch interval associated with the real COB.
12. The computer-readable storage medium of claim 11 , the flushing the stream processor further comprising polling the scoreboard to determine that the flushing is complete.
13. A device comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform operations of:
detecting a real close of books (COB) of a data transport, the real COB associated with a set of raw events transmitted over the data transport and detected after a preconfigured number of validations have occurred for a corresponding batch time, the preconfigured number of validations corresponding to a ratio between the set of raw events transmitted over the data transport and a second set of events written to a raw event storage device;
flushing a stream processor in response to detecting the real COB;
retrieving a set of processed events from a distributed file system after the flushing is complete;
performing a fact COB computation on the set of processed events and the set of raw events, the set of processed events and the set of raw events comprising cogrouping the processed events and the raw events based on a join key, the fact COB computation outputting a number of missing events, each missing event representing a raw event that is not cogrouped with a corresponding processed events; and
certifying the processed events upon determining that the number of missing events is below a threshold.
14. The device of claim 13 , the detecting the real close of books comprising determining that a ratio of the number of events received over the data transport and the number of events written to a raw event storage is above a preconfigured threshold.
15. The device of claim 13 , the operations further comprising identifying a set of missing events based on the cogrouping, the missing events comprising empty cells in a relation generated by the cogrouping.
16. The device of claim 13 , the flushing the stream processor comprising updating a scoreboard indicating with a replay level associated with a batch interval associated with the real COB.
17. The device of claim 16 , the flushing the stream processor further comprising polling the scoreboard to determine that the flushing is complete.