Joining and dimensional annotation in a streaming pipeline
Disclosed are embodiments for providing batch performance using a stream processor. In one embodiment, a method is disclosed comprising receiving, at a stream processor, an event, the stream processor including a plurality of processing stages; generating, by the stream processor, an augmented event based on the event, the augmented event including at least one additional field not appearing in the event, the additional field generated by an operation selected from the group consisting of a join or dimensional annotation operation; and emitting, by the stream processor, the augmented event to downstream consumer.
1. A method comprising:
receiving, at a stream processor, primary event via an input stream;
writing, by the stream processor, the primary event to a join table upon determining that the primary event has not been written to the join table;
querying, by the stream processor, the join table for a secondary event received via the input stream;
if the secondary event is in the join table:
updating, by the stream processor, a status of the secondary event,
generating, by the stream processor, an augmented event, the augmented event comprising fields from both the primary event and the secondary event, and
emitting, by the stream processor, the augmented event to downstream consumer; and
emitting, by the stream processor, the primary event.
2. The method of claim 1 , further comprising:
determining that the secondary event is not in the join table;
receiving the secondary event after emitting the primary event; and
determining that the join table includes the primary event.
3. The method of claim 2 , further comprising labeling the secondary event as an in-order join if the join table includes the primary event.
4. The method of claim 1 , further comprising:
labeling the secondary event as an out-of-order join if the join table includes the secondary event.
5. The method of claim 1 , the generating an augmented event comprising annotating the event with an immutable value, the immutable value stored in a dimension store.
6. The method of claim 5 , further comprising labeling the event with an annotation label based on whether the annotating was successful, a failure, or partially successful.
7. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
receiving, at a stream processor, a primary event via an input stream;
writing, by the stream processor, the primary event to a join table upon determining that the primary event has not been written to the join table;
querying, by the stream processor, the join table for a secondary event received via the input stream;
if the secondary event is in the join table:
updating, by the stream processor, a status of the secondary event,
generating, by the stream processor, an augmented event comprising fields from both the primary event and the secondary event, and
emitting, by the stream processor, the augmented event to downstream consumer; and
emitting, by the stream processor, the primary event.
8. The computer-readable storage medium of claim 7 , the steps further comprising:
determining that the secondary event is not in the join table;
receiving the secondary event after emitting the primary event; and
determining that the join table includes the primary event.
9. The computer-readable storage medium of claim 8 , the steps further comprising labeling the secondary event as an in-order join if the join table includes the primary event.
10. The computer-readable storage medium of claim 8 , the steps further comprising:
labeling the secondary event as an out-of-order join if the join table includes the secondary event.
11. The computer-readable storage medium of claim 7 , the generating an augmented event comprising annotating the event with an immutable value, the immutable value stored in a dimension store.
12. The computer-readable storage medium of claim 11 , the steps further comprising labeling the event with an annotation label based on whether the annotating was successful, a failure, or partially successful.
13. An apparatus comprising:
a processor; and
a storage medium for tangibly storing thereon program logic for execution by the processor, the stored program logic causing the processor to perform operations of:
receiving a primary event via an input stream;
writing the primary event to a join table upon determining that the primary event has not been written to the join table;
querying the join table for a secondary event received via the input stream;
if the secondary event is in the join table:
updating, by the stream processor, a status of the secondary event,
generating an augmented event based on the event, the augmented event comprising fields from both the primary event and the secondary event, and
emitting the augmented event to downstream consumer; and
emitting, by the stream processor, the primary event.
14. The apparatus of claim 13 , the operations further comprising:
determining that the secondary event is not in the join table;
receiving the secondary event after emitting the primary event;
determining that the join table includes the primary event;
labeling the secondary event as an in-order join if the join table includes the primary event.
15. The apparatus of claim 13 , the operations further comprising:
labeling the secondary as an out-of-order join if the join table includes the secondary event.
16. The apparatus of claim 13 , the generating an augmented event comprising annotating the event with an immutable value, the immutable value stored in a dimension store.